-- Logs begin at Sun 2022-09-04 00:10:31 UTC, end at Sun 2022-09-04 01:26:53 UTC. -- Sep 04 00:45:29.682570 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 04 00:45:29.692887 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 04 00:45:29.694958 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: *** Starting uWSGI 2.0.18-debian (64bit) on [Sun Sep 4 00:45:29 2022] *** Sep 04 00:45:29.694958 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 04 00:45:29.694958 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 04 00:45:29.694958 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: nodename: dv-cinder-master-3-iscsi Sep 04 00:45:29.694958 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: machine: x86_64 Sep 04 00:45:29.694958 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: clock source: unix Sep 04 00:45:29.694958 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: pcre jit disabled Sep 04 00:45:29.694958 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: detected number of CPU cores: 4 Sep 04 00:45:29.694958 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: current working directory: / Sep 04 00:45:29.694958 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: detected binary path: /usr/bin/uwsgi-core Sep 04 00:45:29.695792 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: your processes number limit is 47680 Sep 04 00:45:29.695792 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: your memory page size is 4096 bytes Sep 04 00:45:29.695792 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: detected max file descriptor number: 2048 Sep 04 00:45:29.695792 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: lock engine: pthread robust mutexes Sep 04 00:45:29.695792 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: thunder lock: enabled Sep 04 00:45:29.695792 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 04 00:45:29.695792 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 04 00:45:29.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: Python main interpreter initialized at 0x563678b859c0 Sep 04 00:45:29.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: python threads support enabled Sep 04 00:45:29.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: your server socket listen backlog is limited to 100 connections Sep 04 00:45:29.733976 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: your mercy for graceful operations on workers is 90 seconds Sep 04 00:45:29.734288 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: mapped 403077 bytes (393 KB) for 2 cores Sep 04 00:45:29.734364 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: *** Operational MODE: preforking *** Sep 04 00:45:29.734590 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: *** uWSGI is running in multiple interpreter mode *** Sep 04 00:45:29.734590 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: spawned uWSGI master process (pid: 81546) Sep 04 00:45:29.734781 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 04 00:45:29.736054 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: spawned uWSGI worker 1 (pid: 81558, cores: 1) Sep 04 00:45:29.736054 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: spawned uWSGI worker 2 (pid: 81559, cores: 1) Sep 04 00:45:29.736054 dv-cinder-master-3-iscsi devstack@keystone.service[81546]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 04 00:45:34.136361 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 04 00:45:34.136361 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: from cryptography.utils import int_from_bytes Sep 04 00:45:34.137200 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 04 00:45:34.137200 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: from cryptography.utils import int_from_bytes Sep 04 00:45:34.137631 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 04 00:45:34.137631 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: from cryptography.utils import int_from_bytes Sep 04 00:45:34.137734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 04 00:45:34.137734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: from cryptography.utils import int_from_bytes Sep 04 00:45:34.187217 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 04 00:45:34.187382 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 04 00:45:34.187479 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 04 00:45:34.187560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 04 00:45:34.187623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] command line args: [] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 04 00:45:34.187758 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] command line args: [] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 04 00:45:34.187844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 04 00:45:34.188005 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 04 00:45:34.188091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ================================================================================ {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 04 00:45:34.188191 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ================================================================================ {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 04 00:45:34.188270 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] admin_token = **** {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.188446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] admin_token = **** {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.188532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] config_dir = [] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.188673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.188758 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] config_dir = [] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.188861 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] config_source = [] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.188940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.189045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] config_source = [] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.189141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] debug = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.189252 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] debug = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.189331 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.189557 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.189645 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.189715 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.189851 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.189952 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.190061 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.190158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.190296 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.190413 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.190568 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] list_limit = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.190693 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] list_limit = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.190796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] log_config_append = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.190897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] log_config_append = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.190999 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.191109 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.191206 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] log_dir = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.191305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] log_dir = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.191406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] log_file = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.191507 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] log_file = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.191603 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.191688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.191811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.191893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.191971 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.192086 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.192161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.192289 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.192368 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.192485 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.192566 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.192684 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.192809 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.192888 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.193482 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.193619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.193772 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.193871 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.193988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.194051 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.194202 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.194284 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.194437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.194562 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.194676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.194803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.194925 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.195044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.195157 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.195283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.195643 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.42/identity {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.195761 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.42/identity {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.195886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] publish_errors = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.196006 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] publish_errors = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.196119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.196230 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.196349 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.196515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.196666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.196783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.196783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.196997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.197145 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.197262 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.197381 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] standard_threads = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.197502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] standard_threads = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.197616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.197734 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.197859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.197974 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.198093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.198212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.198326 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] use_journal = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.198445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] use_journal = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.198570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] use_json = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.198690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] use_json = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.198793 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] use_stderr = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.198895 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] use_stderr = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.199014 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] use_syslog = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.199129 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] use_syslog = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.199252 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.199373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 04 00:45:34.199493 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.199607 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.199720 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.199839 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.200031 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.200154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.200268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.200387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.200500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.200614 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.200734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.200854 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.200968 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.201096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.201221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.201344 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.201466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.201579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.201693 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.201899 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.202018 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.202144 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.202260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.202373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.202482 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.202607 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.202723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.202848 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.202963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.203075 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.203188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.203301 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.203414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.203538 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.203653 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.203768 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.203887 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.204039 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.204155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.204275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.204390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.204509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.204631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.204746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.204746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.204973 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.205105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.205221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.205333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.205446 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.205569 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.205762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.205880 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.206000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.206000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.206217 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.206337 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.206459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.206573 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.206686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.206799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.206918 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.207032 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.207152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.207273 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.207387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.207499 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.207617 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.207617 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.207864 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.208026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.208149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.208264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.208376 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.208494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.208616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.209338 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.209453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.209453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.209754 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.209883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.210006 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.210105 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.210200 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.210200 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.210404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.210513 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.210610 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.210705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.210799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.210901 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.211004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.211137 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.211253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.211373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.211488 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.211613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.211719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.211839 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.211962 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.212079 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.212201 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.212306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.212418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.212531 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.212644 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.212769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.212891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.213005 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.213136 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.213251 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.213377 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.213485 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.213598 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.213708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.213828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.213932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.214055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.214219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.214335 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.214438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.214549 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.214664 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.214785 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.214901 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.215018 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.215135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.215258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.215374 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.215487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.215487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.215715 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.215829 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.215943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.216056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.216177 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.216177 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.216407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.216523 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.216636 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.216755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.216860 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.216980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.217145 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.217274 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.217391 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.217505 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.217618 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.217804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.217921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.218036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.218149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.218263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.218377 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.218502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.218617 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.218730 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.218843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.218962 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.219077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.219203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.219329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.219446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.219585 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.219689 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.219803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.219917 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.220039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.220155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.220269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.220382 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.220496 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.220609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.220736 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.220852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.220929 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.221007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.221095 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.221168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.221252 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.221326 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.221396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.221467 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.221537 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.221608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.221687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.221760 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.221831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.221902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.221978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.222051 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.222128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.222201 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.222271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.222341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.222411 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.222482 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.222560 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.222633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.222704 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.222775 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.222845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.222921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.223001 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.223073 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.223143 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.223214 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.223283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.223360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.223439 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.223554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.223634 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.223797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.223905 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.224031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] auth.external = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.224113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] auth.external = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.224292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.224385 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.224649 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.224746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.224868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.224969 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.225053 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] auth.password = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.230453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] auth.password = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.230631 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] auth.token = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.230707 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] auth.token = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.230791 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.230866 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.230938 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.231019 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.231093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.231165 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.231236 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.231314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.231393 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.231474 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.231548 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.231626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.231700 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.231778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.231862 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] credential.caching = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.231937 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] credential.caching = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.232010 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.232081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.232197 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.232279 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.232360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.232434 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.232517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.232591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.232663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.232735 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.232814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.232893 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.232967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.233039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.233157 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.233232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.233320 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.233397 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.233469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.233540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.233612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.233683 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.233831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.233908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.233980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.234052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.234124 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.234195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.234321 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.234403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.234476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] federation.caching = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.234554 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] federation.caching = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.234627 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.234699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.234778 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.234852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.234923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.234995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.235066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.235139 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.235217 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.235292 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.235371 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.235444 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.235515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.235586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.235669 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.235742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.235814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.235885 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.235956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.236027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.236106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.236180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.236251 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] identity.caching = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.236361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] identity.caching = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.236485 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.236602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.236687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.236763 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.236835 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.236907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.236979 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.237051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.237205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.237285 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.237390 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.237476 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.237556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.237630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.237710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.237784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.237855 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.237934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.238008 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.238079 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.238158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.238232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.238304 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.238375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.238445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.238524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.238598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.238669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.238735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.238806 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.238877 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.238956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.239030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.239101 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.239172 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.239243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.239321 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.239406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.239484 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.239557 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.239628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.239700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.239771 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.239867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.239953 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.240106 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.240241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.240322 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.240400 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.240481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.240555 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.240627 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.240704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.240786 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.240859 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.240939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.241023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.241113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.241187 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.241258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.241330 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.241414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.241489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.241561 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.241632 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.241703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.241775 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.241845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.241967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.242045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.242045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.242173 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.242245 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.242324 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.242459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.242542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.242703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.242786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.242943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.243031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.243185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.243265 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.243434 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.243511 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.243686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.243796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.243919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.244036 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.244152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.244272 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.244520 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.244611 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.244770 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.244848 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.245009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.245136 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.245278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.245371 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.245519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.245597 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.245750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.246073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.246246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.246333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.246505 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.246583 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.246739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.246817 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.246987 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.247073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.247236 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.247315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.247483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.247561 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.247721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.247807 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.247951 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.248029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.248205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.248308 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.248415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.248532 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.248648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.248763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.248879 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.248992 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.249147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.249265 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.249380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.249495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.249608 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.249722 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.249846 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.249962 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.250078 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.250191 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.250306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.250419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.250541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.250656 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.250770 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.250886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.251000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.251114 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.251236 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.251359 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.251475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.251589 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.251703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.251821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.251945 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.252061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.252175 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.252288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.252402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.252515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.252702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.252820 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.252935 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.253050 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.253195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.253310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.253436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.253553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.253667 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.253780 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.253894 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.254008 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.254131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.254243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.254358 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.254506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.254623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.254721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.254826 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] resource.caching = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.254924 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] resource.caching = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.255020 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.255116 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.255219 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.255316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.255420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.256028 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.256132 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.256271 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.256544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.256657 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.256768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.256868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.256972 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.257069 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.257202 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.257299 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.259078 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.259185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.259360 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] role.caching = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.259439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] role.caching = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.259512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] role.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.259584 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] role.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.259665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.259741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.259813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.259885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.260030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.260132 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.260221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.260296 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.260368 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.260440 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.260513 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.260585 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.260672 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.260753 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.260827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.260905 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.260983 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.261055 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.261163 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.261239 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.261312 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.261385 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.261569 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.261663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.261738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.261810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.261882 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.261954 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.262033 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.262107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.262186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.262258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.262330 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.262403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.262482 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.262556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.262628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.262700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.262795 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.262876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.263034 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.263114 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.263289 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.263368 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.263542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.263621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.263783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.263867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.264016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.264095 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.264248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.264327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.264477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.264604 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.264729 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.264808 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.265015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.265172 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.265263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.265338 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.265472 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.265553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.265708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.265787 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.265947 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] token.caching = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.266029 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.266183 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] token.caching = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.266261 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.266423 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.266502 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.266660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.266739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.266898 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.266976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.267154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.267240 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.267396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.267476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.267633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.267712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.267876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.267962 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.268116 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.268195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.268356 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.268434 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.268591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.268678 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.268826 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.268904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.269058 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.269155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.269330 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.269422 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.269588 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.269675 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.269825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.269913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.270068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.270150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.270334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.270413 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.connection = **** {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.270579 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.270659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.connection = **** {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.270831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.270911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.271093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.271885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.272012 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.272089 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.272169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.272248 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.272345 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.272427 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.272576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.272653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.272815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.272896 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.273052 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.273202 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.273333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.273452 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.273568 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.273682 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.273796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.273911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.274035 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.274152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.274267 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.274381 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.274496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.274655 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.274781 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.274898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.275014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.275129 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.275244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.275358 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.275358 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81559) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 04 00:45:34.275582 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 04 00:45:34.275582 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81558) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 04 00:45:34.311918 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.312074 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.312155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.312309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.312394 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.312549 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.321214 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.321463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.353328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.353482 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.353623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.353706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.420081 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.420540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.469449 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.469684 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.470473 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.470713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.630387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.630739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.630928 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.631147 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.631364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.631579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.636636 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.636873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.637118 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.637348 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.637570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.637789 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.654861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.655093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.655312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.655435 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.655522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.655673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.655756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.655904 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.655987 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.656132 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.656215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.656353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.656572 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.656787 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.667706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.668550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.680402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.680631 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.680798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.680945 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.681031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.681210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.681294 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.681439 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.681536 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.681649 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81558) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.681732 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.681874 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.682093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.682311 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81559) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 04 00:45:34.698303 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.698509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 04 00:45:34.698849 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.698932 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.699049 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 04 00:45:34.699153 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 04 00:45:34.699323 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.699407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.699580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 04 00:45:34.699662 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.699833 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.699925 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.700087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.700215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.700469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.700736 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.710225 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.710497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.710769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.710850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.711053 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.711136 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.711343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.711628 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.718043 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.718645 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.720902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.723495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.723495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.723637 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.723701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.723762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.723870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.723956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.724023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.724085 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.724147 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.724217 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.724905 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.724979 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.725041 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.725131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.725197 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.725272 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.725337 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.725399 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.725399 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.725399 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.746976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.747622 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.748429 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.748715 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.749108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.749475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.753366 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.753647 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.753961 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.754186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.754471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.754754 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.760422 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.760704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.761205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.761497 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.766318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.766612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.766899 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.767162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.767245 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.767450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.769164 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.769298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.769391 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.769469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.769544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.769626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.769702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.769702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.769702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.769702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.784247 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.784526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.784866 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.785174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.788616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.789234 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.791824 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.792126 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.792403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.792701 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.796295 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.797873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.799419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.799910 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.803166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.803538 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.803871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.803968 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.804148 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.804230 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.804431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.804524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.804725 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.804840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.805007 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.805313 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.821457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.821569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.821744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.821830 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.822011 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.822095 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.822275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.822377 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.829104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.829201 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.830640 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.830736 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.832143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.832254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.832417 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.832534 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.835274 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.835367 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.835561 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.835644 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.835846 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.835942 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.836137 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.836221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.836521 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.836605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.847605 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.847702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.847891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.847974 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.848169 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.848266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.848448 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.848532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.848825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.848909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.849135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.849721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.849999 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.850148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.865914 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.866019 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.868804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.868906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.871953 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.872050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.873391 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.873502 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.874853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.874939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.875135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.875218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.875536 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.875622 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.881965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.882128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.883402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.883580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.886272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.886521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.886669 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.886920 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.887021 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.887213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.887296 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.887498 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.894907 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.895195 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.895290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.895530 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.895646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.895838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.902431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81558) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.902708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.902796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81559) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 04 00:45:34.902992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.903074 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.903265 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.903360 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.903548 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.903631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.903826 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.903909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.904102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.904186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.904368 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.904450 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.904638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.904718 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.904914 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.904996 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.905219 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.905303 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.905485 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.905578 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.905904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.905989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.906064 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81558) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.906137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.906287 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81559) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 04 00:45:34.941186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x563678b859c0 pid: 81558 (default app) Sep 04 00:45:34.941375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x563678b859c0 pid: 81559 (default app) Sep 04 00:45:34.944569 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bc180b-d191-49fc-bdea-2fa4465ea06d None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:34.944855 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bc180b-d191-49fc-bdea-2fa4465ea06d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:34.945152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bc180b-d191-49fc-bdea-2fa4465ea06d None None] PATH_INFO: `/v3` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:34.949330 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1/1] 10.222.0.42 () {50 vars in 791 bytes} [Sun Sep 4 00:45:34 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:45:42.488838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46402b8d-1a6a-4028-af8a-e7bee9da7fa9 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:42.489323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46402b8d-1a6a-4028-af8a-e7bee9da7fa9 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:42.489406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46402b8d-1a6a-4028-af8a-e7bee9da7fa9 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:42.494367 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1/2] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:45:42 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:45:42.499849 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ba621b-d440-4db2-b7b1-7cec7aaa7b3c None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:42.500309 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ba621b-d440-4db2-b7b1-7cec7aaa7b3c None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:42.500403 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ba621b-d440-4db2-b7b1-7cec7aaa7b3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:42.571763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: INFO dbcounter [None req-14ba621b-d440-4db2-b7b1-7cec7aaa7b3c None None] Registered counter for database keystone Sep 04 00:45:42.590375 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG oslo_db.sqlalchemy.engines [None req-14ba621b-d440-4db2-b7b1-7cec7aaa7b3c None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81558) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 04 00:45:42.596524 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writer thread running {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 04 00:45:42.778737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG passlib.handlers.bcrypt [None req-14ba621b-d440-4db2-b7b1-7cec7aaa7b3c None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81558) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 04 00:45:42.779050 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG passlib.handlers.bcrypt [None req-14ba621b-d440-4db2-b7b1-7cec7aaa7b3c None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81558) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 04 00:45:42.837461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-14ba621b-d440-4db2-b7b1-7cec7aaa7b3c None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:45:42.839047 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-14ba621b-d440-4db2-b7b1-7cec7aaa7b3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:45:42.965320 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 2/3] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:45:42 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 466 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 04 00:45:43.422348 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46402b8d-1a6a-4028-af8a-e7bee9da7fa9 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:43.495027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: INFO dbcounter [None req-46402b8d-1a6a-4028-af8a-e7bee9da7fa9 None None] Registered counter for database keystone Sep 04 00:45:43.517373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG oslo_db.sqlalchemy.engines [None req-46402b8d-1a6a-4028-af8a-e7bee9da7fa9 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81559) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 04 00:45:43.523609 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writer thread running {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 04 00:45:43.641606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46402b8d-1a6a-4028-af8a-e7bee9da7fa9 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:43.645978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedc3e4a-6685-45f8-97e8-c25838e26542 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:43.646790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedc3e4a-6685-45f8-97e8-c25838e26542 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:43.647073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedc3e4a-6685-45f8-97e8-c25838e26542 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:43.647347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedc3e4a-6685-45f8-97e8-c25838e26542 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:43.655535 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedc3e4a-6685-45f8-97e8-c25838e26542 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:43.714754 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: WARNING keystone.common.rbac_enforcer.enforcer [None req-bedc3e4a-6685-45f8-97e8-c25838e26542 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 04 00:45:43.715196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedc3e4a-6685-45f8-97e8-c25838e26542 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:43.727136 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 2/4] 10.222.0.42 () {58 vars in 1324 bytes} [Sun Sep 4 00:45:43 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 04 00:45:43.733231 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ba621b-d440-4db2-b7b1-7cec7aaa7b3c None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:43.748512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ba621b-d440-4db2-b7b1-7cec7aaa7b3c None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:43.753069 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06967a83-7dc7-489d-b4d2-f4e56cbb0d75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:43.754206 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06967a83-7dc7-489d-b4d2-f4e56cbb0d75 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:43.754485 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06967a83-7dc7-489d-b4d2-f4e56cbb0d75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:43.754763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06967a83-7dc7-489d-b4d2-f4e56cbb0d75 admin admin] PATH_INFO: `/v3/projects/f25649add4b1426590abafbe586fc1fa` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:43.756972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06967a83-7dc7-489d-b4d2-f4e56cbb0d75 admin admin] RBAC: Authorizing `identity:get_project(project_id=f25649add4b1426590abafbe586fc1fa)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:43.809003 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: WARNING keystone.common.rbac_enforcer.enforcer [None req-06967a83-7dc7-489d-b4d2-f4e56cbb0d75 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 04 00:45:43.809369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06967a83-7dc7-489d-b4d2-f4e56cbb0d75 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:43.813916 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 3/5] 10.222.0.42 () {56 vars in 1207 bytes} [Sun Sep 4 00:45:43 2022] GET /identity/v3/projects/f25649add4b1426590abafbe586fc1fa => generated 344 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:45:45.762662 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a37bc8-4999-4b2b-9d99-86c46c59ef43 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:45.763687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a37bc8-4999-4b2b-9d99-86c46c59ef43 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:45.764170 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a37bc8-4999-4b2b-9d99-86c46c59ef43 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:45.768453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 3/6] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:45:45 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:45:45.774546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a0ab46-2b05-43cf-a868-d392375ae6d0 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:45.775340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a0ab46-2b05-43cf-a868-d392375ae6d0 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:45.775617 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a0ab46-2b05-43cf-a868-d392375ae6d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:45.795752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-23a0ab46-2b05-43cf-a868-d392375ae6d0 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:45:45.797296 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-23a0ab46-2b05-43cf-a868-d392375ae6d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:45:45.805695 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 4/7] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:45:45 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 04 00:45:46.072818 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a37bc8-4999-4b2b-9d99-86c46c59ef43 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:46.084948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a37bc8-4999-4b2b-9d99-86c46c59ef43 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:46.089156 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676fba78-4bbf-47de-9c0e-7b5495372a00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:46.089756 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676fba78-4bbf-47de-9c0e-7b5495372a00 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:46.090038 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676fba78-4bbf-47de-9c0e-7b5495372a00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:46.090312 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676fba78-4bbf-47de-9c0e-7b5495372a00 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:46.097876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676fba78-4bbf-47de-9c0e-7b5495372a00 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:46.100184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676fba78-4bbf-47de-9c0e-7b5495372a00 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:46.108748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 4/8] 10.222.0.42 () {58 vars in 1324 bytes} [Sun Sep 4 00:45:46 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 04 00:45:46.113914 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a0ab46-2b05-43cf-a868-d392375ae6d0 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:46.123817 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a0ab46-2b05-43cf-a868-d392375ae6d0 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:46.128068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f45ce6-3ab9-4184-ba9b-3c09a01d68c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:46.128685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f45ce6-3ab9-4184-ba9b-3c09a01d68c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:46.128960 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f45ce6-3ab9-4184-ba9b-3c09a01d68c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:46.129276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f45ce6-3ab9-4184-ba9b-3c09a01d68c9 admin admin] PATH_INFO: `/v3/users/01782590dd294bd488a6d5c08a043ce1` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:46.130704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f45ce6-3ab9-4184-ba9b-3c09a01d68c9 admin admin] RBAC: Authorizing `identity:get_user(user_id=01782590dd294bd488a6d5c08a043ce1)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:46.132522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f45ce6-3ab9-4184-ba9b-3c09a01d68c9 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:46.134061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 5/9] 10.222.0.42 () {56 vars in 1198 bytes} [Sun Sep 4 00:45:46 2022] GET /identity/v3/users/01782590dd294bd488a6d5c08a043ce1 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:45:48.113928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef5e46c-db78-4d68-98fc-3175ab754cf7 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:48.114470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef5e46c-db78-4d68-98fc-3175ab754cf7 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:48.114470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef5e46c-db78-4d68-98fc-3175ab754cf7 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:48.116485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 5/10] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:45:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:45:48.121904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a527cdd5-fa32-4461-a6c8-6c1d1eb79a94 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:48.122320 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a527cdd5-fa32-4461-a6c8-6c1d1eb79a94 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:48.122461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a527cdd5-fa32-4461-a6c8-6c1d1eb79a94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:48.140969 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-a527cdd5-fa32-4461-a6c8-6c1d1eb79a94 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:45:48.142185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a527cdd5-fa32-4461-a6c8-6c1d1eb79a94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:45:48.183459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9de8c1-2c8d-4fde-adcc-e3169f84bee6 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:48.183739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9de8c1-2c8d-4fde-adcc-e3169f84bee6 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:48.184200 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9de8c1-2c8d-4fde-adcc-e3169f84bee6 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:48.185551 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 6/11] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:45:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:45:48.190702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121e7899-02ff-41b3-a12d-34dafd9f9ea4 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:48.190979 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121e7899-02ff-41b3-a12d-34dafd9f9ea4 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:48.191291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121e7899-02ff-41b3-a12d-34dafd9f9ea4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:48.223105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 6/12] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:45:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:45:48.232893 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG passlib.handlers.bcrypt [None req-121e7899-02ff-41b3-a12d-34dafd9f9ea4 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81559) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 04 00:45:48.233317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG passlib.handlers.bcrypt [None req-121e7899-02ff-41b3-a12d-34dafd9f9ea4 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81559) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 04 00:45:48.282887 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-121e7899-02ff-41b3-a12d-34dafd9f9ea4 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:45:48.284439 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-121e7899-02ff-41b3-a12d-34dafd9f9ea4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:45:48.383023 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 7/13] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:45:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:45:48.492263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a527cdd5-fa32-4461-a6c8-6c1d1eb79a94 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:48.517903 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:48.597207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a527cdd5-fa32-4461-a6c8-6c1d1eb79a94 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:48.629835 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3206db-79f5-4295-ab78-d397b6ef0729 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:48.631045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3206db-79f5-4295-ab78-d397b6ef0729 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:48.631384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3206db-79f5-4295-ab78-d397b6ef0729 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:48.631659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3206db-79f5-4295-ab78-d397b6ef0729 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:48.642838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3206db-79f5-4295-ab78-d397b6ef0729 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:45:48.650368 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121e7899-02ff-41b3-a12d-34dafd9f9ea4 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:48.652119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-0a3206db-79f5-4295-ab78-d397b6ef0729 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:45:48.652119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:45:48.652119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:45:48.652119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:45:48.652119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:45:48.652119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:45:48.652119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:45:48.652119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:45:48.652119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:45:48.652119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:45:48.652119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:45:48.652119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:45:48.652119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:45:48.652119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:45:48.652119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:45:48.652119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:45:48.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:45:48.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:45:48.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:45:48.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:45:48.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:45:48.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:45:48.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:45:48.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:45:48.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:45:48.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:45:48.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:45:48.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:45:48.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:45:48.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:45:48.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:45:48.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:45:48.653919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:45:48.653919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:45:48.653919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:45:48.653919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:45:48.653919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:45:48.653919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:45:48.653919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:45:48.653919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:45:48.653919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:45:48.653919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:45:48.653919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:45:48.653919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:45:48.653919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 7/14] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:45:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:45:48.658850 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3206db-79f5-4295-ab78-d397b6ef0729 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:48.740465 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:48.750023 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121e7899-02ff-41b3-a12d-34dafd9f9ea4 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:48.752487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:48.758570 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3206db-79f5-4295-ab78-d397b6ef0729 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:48.789045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37f7310-5489-40f8-bd9f-bc0d5404b685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:48.790165 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37f7310-5489-40f8-bd9f-bc0d5404b685 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:48.790451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37f7310-5489-40f8-bd9f-bc0d5404b685 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:48.790723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37f7310-5489-40f8-bd9f-bc0d5404b685 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:48.802524 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15f16f9-1810-41aa-b39e-607d2cc454b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:48.803416 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15f16f9-1810-41aa-b39e-607d2cc454b4 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:48.803901 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15f16f9-1810-41aa-b39e-607d2cc454b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:48.804322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15f16f9-1810-41aa-b39e-607d2cc454b4 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:48.805380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e15f16f9-1810-41aa-b39e-607d2cc454b4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:45:48.805832 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e15f16f9-1810-41aa-b39e-607d2cc454b4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:48.807458 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e15f16f9-1810-41aa-b39e-607d2cc454b4 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:48.822370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 8/15] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:45:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:45:48.827919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15f16f9-1810-41aa-b39e-607d2cc454b4 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:48.859030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37f7310-5489-40f8-bd9f-bc0d5404b685 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:48.860541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37f7310-5489-40f8-bd9f-bc0d5404b685 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:48.926736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15f16f9-1810-41aa-b39e-607d2cc454b4 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:48.957260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 8/16] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:45:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 04 00:45:48.961419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc553b1-5971-4501-bb0b-e942cb2814c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:48.962052 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc553b1-5971-4501-bb0b-e942cb2814c9 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:48.962369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc553b1-5971-4501-bb0b-e942cb2814c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:48.962630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc553b1-5971-4501-bb0b-e942cb2814c9 None admin] PATH_INFO: `/v3/users/01782590dd294bd488a6d5c08a043ce1` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:48.963458 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37f7310-5489-40f8-bd9f-bc0d5404b685 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:48.963743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc553b1-5971-4501-bb0b-e942cb2814c9 None admin] RBAC: Authorizing `identity:get_user(user_id=01782590dd294bd488a6d5c08a043ce1)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:48.965243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc553b1-5971-4501-bb0b-e942cb2814c9 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:48.966350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:48.966951 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 9/17] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:45:48 2022] GET /identity/v3/users/01782590dd294bd488a6d5c08a043ce1 => generated 247 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:45:48.972134 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc553b1-5971-4501-bb0b-e942cb2814c9 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:48.987425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:49.062386 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37f7310-5489-40f8-bd9f-bc0d5404b685 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:49.071179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc553b1-5971-4501-bb0b-e942cb2814c9 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:49.099091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4702a1-7c4d-4ccb-8803-b2963b0962d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:49.100471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4702a1-7c4d-4ccb-8803-b2963b0962d9 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:49.100743 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4702a1-7c4d-4ccb-8803-b2963b0962d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:49.101013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4702a1-7c4d-4ccb-8803-b2963b0962d9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:49.108554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9861c75d-32c6-4538-8454-1906c74096dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:49.109358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9861c75d-32c6-4538-8454-1906c74096dd None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:49.109636 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9861c75d-32c6-4538-8454-1906c74096dd None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:49.109904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9861c75d-32c6-4538-8454-1906c74096dd None admin] PATH_INFO: `/v3/domains/default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:49.111026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9861c75d-32c6-4538-8454-1906c74096dd None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:49.112473 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9861c75d-32c6-4538-8454-1906c74096dd None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:49.114052 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 10/18] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:45:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:45:49.120098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9861c75d-32c6-4538-8454-1906c74096dd None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:49.120215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4702a1-7c4d-4ccb-8803-b2963b0962d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:49.121966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4702a1-7c4d-4ccb-8803-b2963b0962d9 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:49.140989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-0d4702a1-7c4d-4ccb-8803-b2963b0962d9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:45:49.140989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:45:49.140989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:45:49.140989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:45:49.140989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:45:49.140989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:45:49.140989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:45:49.140989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:45:49.140989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:45:49.140989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:45:49.140989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:45:49.140989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:45:49.140989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:45:49.140989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:45:49.140989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:45:49.140989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:45:49.141902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:45:49.141902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:45:49.141902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:45:49.141902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:45:49.141902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:45:49.141902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:45:49.141902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:45:49.141902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:45:49.141902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:45:49.141902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:45:49.141902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:45:49.141902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:45:49.141902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:45:49.141902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:45:49.141902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:45:49.141902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:45:49.141902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:45:49.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:45:49.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:45:49.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:45:49.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 9/19] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:45:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:45:49.148540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4702a1-7c4d-4ccb-8803-b2963b0962d9 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:49.202285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:49.206623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:49.221178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9861c75d-32c6-4538-8454-1906c74096dd None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:49.250384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4702a1-7c4d-4ccb-8803-b2963b0962d9 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:49.255708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171424e8-8e7f-4a49-a353-e7b750474ef2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:49.256314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171424e8-8e7f-4a49-a353-e7b750474ef2 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:49.256590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171424e8-8e7f-4a49-a353-e7b750474ef2 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:49.256859 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171424e8-8e7f-4a49-a353-e7b750474ef2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:49.257620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171424e8-8e7f-4a49-a353-e7b750474ef2 None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.domain.id=default, user.id=01782590dd294bd488a6d5c08a043ce1) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:45:49.257922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171424e8-8e7f-4a49-a353-e7b750474ef2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:49.259357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171424e8-8e7f-4a49-a353-e7b750474ef2 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:49.265915 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 11/20] 10.222.0.42 () {56 vars in 1322 bytes} [Sun Sep 4 00:45:49 2022] GET /identity/v3/role_assignments?user.id=01782590dd294bd488a6d5c08a043ce1&scope.domain.id=default&role.id=19c006412bec45dc8febd7cc86b9678f => generated 233 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:45:49.285896 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffbeb1c-7fb3-49dc-bf40-9a009f57289f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:49.287130 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffbeb1c-7fb3-49dc-bf40-9a009f57289f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:49.287668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffbeb1c-7fb3-49dc-bf40-9a009f57289f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:49.287946 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffbeb1c-7fb3-49dc-bf40-9a009f57289f None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:49.288633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffbeb1c-7fb3-49dc-bf40-9a009f57289f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:45:49.288933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffbeb1c-7fb3-49dc-bf40-9a009f57289f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:49.290428 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffbeb1c-7fb3-49dc-bf40-9a009f57289f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:49.310434 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 10/21] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:45:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:45:51.400662 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017c205c-c021-49fa-b54e-d49893a6d10b None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:51.401728 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017c205c-c021-49fa-b54e-d49893a6d10b None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:51.402224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017c205c-c021-49fa-b54e-d49893a6d10b None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:51.403775 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 12/22] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:45:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:45:51.408988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d897053-30ea-4ba6-95c7-a2aecce19377 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:51.409847 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d897053-30ea-4ba6-95c7-a2aecce19377 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:51.410341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d897053-30ea-4ba6-95c7-a2aecce19377 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:51.439357 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-6d897053-30ea-4ba6-95c7-a2aecce19377 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:45:51.441610 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6d897053-30ea-4ba6-95c7-a2aecce19377 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:45:51.526892 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 11/23] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:45:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:45:51.676072 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41f3c3a-44dd-4dcd-b768-4817a2c6f65d None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:51.676690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41f3c3a-44dd-4dcd-b768-4817a2c6f65d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:51.677250 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41f3c3a-44dd-4dcd-b768-4817a2c6f65d None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:51.678837 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 13/24] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:45:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:45:51.684562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80525085-65e1-4f54-b560-22c36200af68 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:51.685070 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80525085-65e1-4f54-b560-22c36200af68 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:51.685610 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80525085-65e1-4f54-b560-22c36200af68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:51.714116 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-80525085-65e1-4f54-b560-22c36200af68 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:45:51.715691 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-80525085-65e1-4f54-b560-22c36200af68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:45:51.773495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:51.797751 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 12/25] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:45:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:45:51.875922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae5fcd8-b24b-44ac-bbe3-203a69ffe1df None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:51.876549 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae5fcd8-b24b-44ac-bbe3-203a69ffe1df None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:51.877049 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae5fcd8-b24b-44ac-bbe3-203a69ffe1df None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:51.878621 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 14/26] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:45:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:45:51.885118 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a211540e-74b2-40d3-96eb-9c8246be7ca7 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:51.885668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a211540e-74b2-40d3-96eb-9c8246be7ca7 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:51.886146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a211540e-74b2-40d3-96eb-9c8246be7ca7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:51.908868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-a211540e-74b2-40d3-96eb-9c8246be7ca7 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:45:51.910431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a211540e-74b2-40d3-96eb-9c8246be7ca7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:45:51.988885 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 13/27] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:45:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:45:52.005173 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca6ac1b-d370-450f-9391-553901a3ea55 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:52.006294 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca6ac1b-d370-450f-9391-553901a3ea55 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:52.006824 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca6ac1b-d370-450f-9391-553901a3ea55 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:52.008356 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 15/28] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:45:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:45:52.013664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb36d47-b1d3-405c-bd27-7fb0a328eb7a None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:52.014175 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb36d47-b1d3-405c-bd27-7fb0a328eb7a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:52.014668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb36d47-b1d3-405c-bd27-7fb0a328eb7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:52.033997 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-2fb36d47-b1d3-405c-bd27-7fb0a328eb7a None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:45:52.035506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2fb36d47-b1d3-405c-bd27-7fb0a328eb7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:45:52.084853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca6ac1b-d370-450f-9391-553901a3ea55 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:52.121831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 14/29] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:45:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:45:52.137695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb36d47-b1d3-405c-bd27-7fb0a328eb7a None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:52.185471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:52.266500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca6ac1b-d370-450f-9391-553901a3ea55 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:52.304127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c48596-8521-4690-aaad-6330257cd917 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:52.305169 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c48596-8521-4690-aaad-6330257cd917 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:52.305654 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c48596-8521-4690-aaad-6330257cd917 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:52.305901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb36d47-b1d3-405c-bd27-7fb0a328eb7a None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:52.306359 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c48596-8521-4690-aaad-6330257cd917 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:52.325114 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c48596-8521-4690-aaad-6330257cd917 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:45:52.330502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-00c48596-8521-4690-aaad-6330257cd917 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:45:52.330502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:45:52.330502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:45:52.330502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:45:52.330502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:45:52.330502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:45:52.330502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:45:52.330502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:45:52.330502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:45:52.330502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:45:52.330502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:45:52.330502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:45:52.330502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:45:52.330502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:45:52.330502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:45:52.330502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:45:52.331441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:45:52.331441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:45:52.331441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:45:52.331441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:45:52.331441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:45:52.331441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:45:52.331441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:45:52.331441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:45:52.331441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:45:52.331441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:45:52.331441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:45:52.331441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:45:52.331441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:45:52.331441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:45:52.331441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:45:52.331441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:45:52.332166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:45:52.332166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:45:52.332166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:45:52.332166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:45:52.332166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:45:52.332166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:45:52.332166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:45:52.332166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:45:52.332166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:45:52.332166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:45:52.332166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:45:52.332166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:45:52.339842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 16/30] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:45:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:45:52.345699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c48596-8521-4690-aaad-6330257cd917 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:52.358084 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:52.389761 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1de979-37b6-48da-aaba-fff020abac06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:52.390391 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1de979-37b6-48da-aaba-fff020abac06 None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:52.390669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1de979-37b6-48da-aaba-fff020abac06 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:52.390938 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1de979-37b6-48da-aaba-fff020abac06 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:52.391679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1de979-37b6-48da-aaba-fff020abac06 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:52.393166 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1de979-37b6-48da-aaba-fff020abac06 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:52.488994 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c48596-8521-4690-aaad-6330257cd917 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:52.539780 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd526a76-31a2-4202-8b95-f228508c4ac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:52.540733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd526a76-31a2-4202-8b95-f228508c4ac5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:52.541303 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd526a76-31a2-4202-8b95-f228508c4ac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:52.541787 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd526a76-31a2-4202-8b95-f228508c4ac5 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:52.543637 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd526a76-31a2-4202-8b95-f228508c4ac5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:45:52.544163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd526a76-31a2-4202-8b95-f228508c4ac5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:52.545844 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd526a76-31a2-4202-8b95-f228508c4ac5 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:52.567182 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 17/31] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:45:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:45:52.568865 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd526a76-31a2-4202-8b95-f228508c4ac5 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:52.634612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 15/32] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Sep 4 00:45:52 2022] POST /identity/v3/roles => generated 219 bytes in 497 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 00:45:52.636245 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1de979-37b6-48da-aaba-fff020abac06 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:52.637108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:52.707942 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd526a76-31a2-4202-8b95-f228508c4ac5 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:52.796210 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8d3fd0-44c4-48e6-afec-da0f77c83308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:52.797133 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8d3fd0-44c4-48e6-afec-da0f77c83308 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:52.797584 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8d3fd0-44c4-48e6-afec-da0f77c83308 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:52.798007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8d3fd0-44c4-48e6-afec-da0f77c83308 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:52.816137 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8d3fd0-44c4-48e6-afec-da0f77c83308 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:52.819947 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:52.820969 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8d3fd0-44c4-48e6-afec-da0f77c83308 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:52.826712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1de979-37b6-48da-aaba-fff020abac06 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:52.845023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-ce8d3fd0-44c4-48e6-afec-da0f77c83308 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:45:52.845023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:45:52.845023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:45:52.845023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:45:52.845023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:45:52.845023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:45:52.845023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:45:52.845023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:45:52.845023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:45:52.845023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:45:52.845023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:45:52.845023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:45:52.845023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:45:52.845023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:45:52.845023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:45:52.845023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:45:52.845929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:45:52.845929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:45:52.845929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:45:52.845929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:45:52.845929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:45:52.845929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:45:52.845929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:45:52.845929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:45:52.845929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:45:52.845929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:45:52.845929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:45:52.845929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:45:52.845929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:45:52.845929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:45:52.845929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:45:52.845929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:45:52.845929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:45:52.846752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:45:52.846752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:45:52.846752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:45:52.849861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 18/33] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:45:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:45:52.850990 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8d3fd0-44c4-48e6-afec-da0f77c83308 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:52.873931 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6863bfd-9c5a-49e8-b410-1139ff094aae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:52.874473 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6863bfd-9c5a-49e8-b410-1139ff094aae None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:52.874761 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6863bfd-9c5a-49e8-b410-1139ff094aae None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:52.875041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6863bfd-9c5a-49e8-b410-1139ff094aae None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:52.875728 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6863bfd-9c5a-49e8-b410-1139ff094aae None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:52.877315 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6863bfd-9c5a-49e8-b410-1139ff094aae None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:52.954987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 16/34] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Sep 4 00:45:52 2022] POST /identity/v3/roles => generated 225 bytes in 319 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:45:52.955689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6863bfd-9c5a-49e8-b410-1139ff094aae None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:53.023191 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8d3fd0-44c4-48e6-afec-da0f77c83308 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:53.078862 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:53.111993 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6fbd27-f104-4f5d-b06f-177c571889ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:53.112616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6fbd27-f104-4f5d-b06f-177c571889ca None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:53.112898 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6fbd27-f104-4f5d-b06f-177c571889ca None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:53.113205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6fbd27-f104-4f5d-b06f-177c571889ca None admin] PATH_INFO: `/v3/users/01782590dd294bd488a6d5c08a043ce1` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:53.117106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6fbd27-f104-4f5d-b06f-177c571889ca None admin] RBAC: Authorizing `identity:get_user(user_id=01782590dd294bd488a6d5c08a043ce1)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:53.117106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6fbd27-f104-4f5d-b06f-177c571889ca None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:53.120370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 19/35] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:45:52 2022] GET /identity/v3/users/01782590dd294bd488a6d5c08a043ce1 => generated 247 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:45:53.124765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6fbd27-f104-4f5d-b06f-177c571889ca None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:53.136593 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6863bfd-9c5a-49e8-b410-1139ff094aae None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:53.176366 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a94dc7-6db7-4cd1-9b75-71aa906c1ced None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:53.177276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a94dc7-6db7-4cd1-9b75-71aa906c1ced None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:53.177726 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a94dc7-6db7-4cd1-9b75-71aa906c1ced None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:53.178146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a94dc7-6db7-4cd1-9b75-71aa906c1ced None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:53.178952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a94dc7-6db7-4cd1-9b75-71aa906c1ced None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:45:53.179424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a94dc7-6db7-4cd1-9b75-71aa906c1ced None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:53.180956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a94dc7-6db7-4cd1-9b75-71aa906c1ced None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:53.195781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 17/36] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:45:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:45:53.201901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a94dc7-6db7-4cd1-9b75-71aa906c1ced None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:53.224204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:53.257958 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6fbd27-f104-4f5d-b06f-177c571889ca None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:53.297287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f05a27-0b41-4edb-be5e-dd5bb2b8fa89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:53.297877 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f05a27-0b41-4edb-be5e-dd5bb2b8fa89 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:53.298159 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f05a27-0b41-4edb-be5e-dd5bb2b8fa89 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:53.298432 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f05a27-0b41-4edb-be5e-dd5bb2b8fa89 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:53.299618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f05a27-0b41-4edb-be5e-dd5bb2b8fa89 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:53.301054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f05a27-0b41-4edb-be5e-dd5bb2b8fa89 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:53.303189 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 20/37] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:45:53 2022] GET /identity/v3/domains/default => generated 199 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:45:53.309300 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f05a27-0b41-4edb-be5e-dd5bb2b8fa89 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:53.317213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a94dc7-6db7-4cd1-9b75-71aa906c1ced None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:53.364359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7698e3-462f-42d8-8d3e-b091b8d741f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:53.365745 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7698e3-462f-42d8-8d3e-b091b8d741f0 None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:53.366457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7698e3-462f-42d8-8d3e-b091b8d741f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:53.367150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7698e3-462f-42d8-8d3e-b091b8d741f0 None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:53.368338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7698e3-462f-42d8-8d3e-b091b8d741f0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:53.370114 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7698e3-462f-42d8-8d3e-b091b8d741f0 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:53.372267 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:53.420801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f05a27-0b41-4edb-be5e-dd5bb2b8fa89 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:53.456954 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f6d501-2988-42bc-978a-c325f162dd4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:53.457659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f6d501-2988-42bc-978a-c325f162dd4e None admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:53.458048 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f6d501-2988-42bc-978a-c325f162dd4e None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:53.458994 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f6d501-2988-42bc-978a-c325f162dd4e None admin] PATH_INFO: `/v3/domains/default/users/01782590dd294bd488a6d5c08a043ce1/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:53.464001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f6d501-2988-42bc-978a-c325f162dd4e None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=01782590dd294bd488a6d5c08a043ce1, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:53.469278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 18/38] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 4 00:45:53 2022] POST /identity/v3/projects => generated 301 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 00:45:53.469450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f6d501-2988-42bc-978a-c325f162dd4e None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:53.469565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da92163-df63-4261-9081-ac010d4403d3 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:53.469565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da92163-df63-4261-9081-ac010d4403d3 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:53.469565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da92163-df63-4261-9081-ac010d4403d3 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:53.469565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 19/39] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:45:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:45:53.475516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda4271d-2a91-4e78-b531-ff402ba4fc16 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:53.476409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda4271d-2a91-4e78-b531-ff402ba4fc16 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:53.476703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda4271d-2a91-4e78-b531-ff402ba4fc16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:53.498960 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-dda4271d-2a91-4e78-b531-ff402ba4fc16 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:45:53.500570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-dda4271d-2a91-4e78-b531-ff402ba4fc16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:45:53.576786 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:53.583054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 20/40] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:45:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:45:53.585175 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dce7511-50a0-4f08-bec0-d6fb4f387519 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:53.585653 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dce7511-50a0-4f08-bec0-d6fb4f387519 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:53.586085 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dce7511-50a0-4f08-bec0-d6fb4f387519 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:53.587667 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 21/41] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:45:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:45:53.593496 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46990fa1-9874-49ab-a7c1-618ee825fca3 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:53.594032 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46990fa1-9874-49ab-a7c1-618ee825fca3 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:53.594475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46990fa1-9874-49ab-a7c1-618ee825fca3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:53.617014 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-46990fa1-9874-49ab-a7c1-618ee825fca3 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:45:53.618379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-46990fa1-9874-49ab-a7c1-618ee825fca3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:45:53.657830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 21/42] 10.222.0.42 () {58 vars in 1361 bytes} [Sun Sep 4 00:45:53 2022] PUT /identity/v3/domains/default/users/01782590dd294bd488a6d5c08a043ce1/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 00:45:53.696749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 22/43] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:45:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:45:53.877368 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f6d501-2988-42bc-978a-c325f162dd4e None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:53.986399 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46990fa1-9874-49ab-a7c1-618ee825fca3 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:54.012627 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f6d501-2988-42bc-978a-c325f162dd4e None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:54.029170 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:54.054166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea970cf2-16f1-4bc5-9bdf-ad6fea5581ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:54.054808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea970cf2-16f1-4bc5-9bdf-ad6fea5581ea None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:54.055491 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea970cf2-16f1-4bc5-9bdf-ad6fea5581ea None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:54.055779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea970cf2-16f1-4bc5-9bdf-ad6fea5581ea None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:54.056487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea970cf2-16f1-4bc5-9bdf-ad6fea5581ea None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:54.058149 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea970cf2-16f1-4bc5-9bdf-ad6fea5581ea None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:54.124375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46990fa1-9874-49ab-a7c1-618ee825fca3 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:54.161224 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:54.170501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd17930-b217-46e4-8d14-b9c7986b5fbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:54.171173 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd17930-b217-46e4-8d14-b9c7986b5fbf None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:54.171523 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd17930-b217-46e4-8d14-b9c7986b5fbf None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:54.171785 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd17930-b217-46e4-8d14-b9c7986b5fbf None admin] PATH_INFO: `/v3/domains/default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:54.173048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd17930-b217-46e4-8d14-b9c7986b5fbf None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:54.174803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd17930-b217-46e4-8d14-b9c7986b5fbf None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:54.176746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 23/44] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:45:53 2022] GET /identity/v3/domains/default => generated 199 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:45:54.183813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd17930-b217-46e4-8d14-b9c7986b5fbf None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:54.289132 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 22/45] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Sep 4 00:45:53 2022] POST /identity/v3/roles => generated 223 bytes in 411 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:45:54.321384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd17930-b217-46e4-8d14-b9c7986b5fbf None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:54.372275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29f1519-dbc6-4cb3-ae10-f458472f2319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:54.372908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29f1519-dbc6-4cb3-ae10-f458472f2319 None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:54.373224 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29f1519-dbc6-4cb3-ae10-f458472f2319 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:54.373500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29f1519-dbc6-4cb3-ae10-f458472f2319 None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:54.374706 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29f1519-dbc6-4cb3-ae10-f458472f2319 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:54.375976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29f1519-dbc6-4cb3-ae10-f458472f2319 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:54.468813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 24/46] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 4 00:45:54 2022] POST /identity/v3/projects => generated 312 bytes in 286 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:45:55.600167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed88a078-62ce-4f26-ae6f-d4815ddc83d3 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:55.600683 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed88a078-62ce-4f26-ae6f-d4815ddc83d3 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:55.600683 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed88a078-62ce-4f26-ae6f-d4815ddc83d3 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:55.602233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 23/47] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:45:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:45:55.607439 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d02b3e2-0847-41a1-9c62-a2c49cbe5546 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:55.607912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d02b3e2-0847-41a1-9c62-a2c49cbe5546 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:55.608114 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d02b3e2-0847-41a1-9c62-a2c49cbe5546 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:55.731417 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-5d02b3e2-0847-41a1-9c62-a2c49cbe5546 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:45:55.732545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-5d02b3e2-0847-41a1-9c62-a2c49cbe5546 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:45:55.774933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:55.797339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 25/48] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:45:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:45:56.064968 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed88a078-62ce-4f26-ae6f-d4815ddc83d3 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:56.165469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed88a078-62ce-4f26-ae6f-d4815ddc83d3 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:56.198559 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c73a92f-7176-4b5f-bf3d-548f72f8e44f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:56.199192 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c73a92f-7176-4b5f-bf3d-548f72f8e44f None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:56.199476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c73a92f-7176-4b5f-bf3d-548f72f8e44f None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:56.199755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c73a92f-7176-4b5f-bf3d-548f72f8e44f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:56.210249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c73a92f-7176-4b5f-bf3d-548f72f8e44f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:45:56.215165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-4c73a92f-7176-4b5f-bf3d-548f72f8e44f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:45:56.215165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:45:56.215165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:45:56.215165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:45:56.215165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:45:56.215165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:45:56.215165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:45:56.215165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:45:56.215165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:45:56.215165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:45:56.215165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:45:56.215165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:45:56.215165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:45:56.215165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:45:56.215165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:45:56.215165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:45:56.216044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:45:56.216044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:45:56.216044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:45:56.216044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:45:56.216044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:45:56.216044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:45:56.216044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:45:56.216044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:45:56.216044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:45:56.216044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:45:56.216044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:45:56.216044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:45:56.216044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:45:56.216044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:45:56.216044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:45:56.216044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:45:56.216786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:45:56.216786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:45:56.216786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:45:56.216786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:45:56.216786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:45:56.216786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:45:56.216786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:45:56.216786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:45:56.216786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:45:56.216786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:45:56.216786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:45:56.216786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:45:56.216786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 24/49] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:45:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:45:56.222223 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d02b3e2-0847-41a1-9c62-a2c49cbe5546 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:56.318938 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d02b3e2-0847-41a1-9c62-a2c49cbe5546 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:56.352654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41ecd79-13af-495c-a0b2-45606f6e8f8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:56.353292 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41ecd79-13af-495c-a0b2-45606f6e8f8d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:56.353679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41ecd79-13af-495c-a0b2-45606f6e8f8d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:56.353956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41ecd79-13af-495c-a0b2-45606f6e8f8d None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:56.354635 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41ecd79-13af-495c-a0b2-45606f6e8f8d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:45:56.354942 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41ecd79-13af-495c-a0b2-45606f6e8f8d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:56.356328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41ecd79-13af-495c-a0b2-45606f6e8f8d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:56.367959 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 26/50] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:45:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:45:56.374440 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c73a92f-7176-4b5f-bf3d-548f72f8e44f None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:56.408333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:56.459213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab543d0-52c3-4fe6-b6bb-9012767f5d56 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:56.459513 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab543d0-52c3-4fe6-b6bb-9012767f5d56 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:56.459770 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab543d0-52c3-4fe6-b6bb-9012767f5d56 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:56.461277 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 27/51] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:45:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:45:56.466245 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302fe7db-948a-45bc-a3af-4b878713f235 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:56.466542 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302fe7db-948a-45bc-a3af-4b878713f235 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:56.466819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302fe7db-948a-45bc-a3af-4b878713f235 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:56.478087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c73a92f-7176-4b5f-bf3d-548f72f8e44f None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:56.484904 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-302fe7db-948a-45bc-a3af-4b878713f235 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:45:56.486124 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-302fe7db-948a-45bc-a3af-4b878713f235 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:45:56.511779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61e5ee6-add1-42d3-9caf-cf43d7eb778e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:56.512416 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61e5ee6-add1-42d3-9caf-cf43d7eb778e None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:56.512697 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61e5ee6-add1-42d3-9caf-cf43d7eb778e None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:56.512982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61e5ee6-add1-42d3-9caf-cf43d7eb778e None admin] PATH_INFO: `/v3/users/01782590dd294bd488a6d5c08a043ce1` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:56.514150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61e5ee6-add1-42d3-9caf-cf43d7eb778e None admin] RBAC: Authorizing `identity:get_user(user_id=01782590dd294bd488a6d5c08a043ce1)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:56.515598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61e5ee6-add1-42d3-9caf-cf43d7eb778e None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:56.517205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 25/52] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:45:56 2022] GET /identity/v3/users/01782590dd294bd488a6d5c08a043ce1 => generated 247 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:45:56.520520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:56.523436 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61e5ee6-add1-42d3-9caf-cf43d7eb778e None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:56.554615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 28/53] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:45:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:45:56.622416 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61e5ee6-add1-42d3-9caf-cf43d7eb778e None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:56.623838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:56.656458 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870e3d8c-5a00-4b54-9779-695a7d4fb380 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:56.657068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870e3d8c-5a00-4b54-9779-695a7d4fb380 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:56.657371 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870e3d8c-5a00-4b54-9779-695a7d4fb380 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:56.657645 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870e3d8c-5a00-4b54-9779-695a7d4fb380 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:56.658843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870e3d8c-5a00-4b54-9779-695a7d4fb380 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:56.660279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870e3d8c-5a00-4b54-9779-695a7d4fb380 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:56.662253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 26/54] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:45:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:45:56.668230 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302fe7db-948a-45bc-a3af-4b878713f235 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:56.768718 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302fe7db-948a-45bc-a3af-4b878713f235 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:56.804306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98eaf0a-5443-4bde-b162-280c55bd17fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:56.804909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98eaf0a-5443-4bde-b162-280c55bd17fa None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:56.805226 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98eaf0a-5443-4bde-b162-280c55bd17fa None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:56.805503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98eaf0a-5443-4bde-b162-280c55bd17fa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:56.806223 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98eaf0a-5443-4bde-b162-280c55bd17fa None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.domain.id=default, user.id=01782590dd294bd488a6d5c08a043ce1) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:45:56.806533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98eaf0a-5443-4bde-b162-280c55bd17fa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:56.807956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98eaf0a-5443-4bde-b162-280c55bd17fa None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:56.816383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 29/55] 10.222.0.42 () {56 vars in 1322 bytes} [Sun Sep 4 00:45:56 2022] GET /identity/v3/role_assignments?user.id=01782590dd294bd488a6d5c08a043ce1&scope.domain.id=default&role.id=19c006412bec45dc8febd7cc86b9678f => generated 530 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:45:56.821543 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870e3d8c-5a00-4b54-9779-695a7d4fb380 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:56.930805 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870e3d8c-5a00-4b54-9779-695a7d4fb380 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:56.967369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a9270d-5034-4866-b8ad-d429b6e2b2c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:56.968010 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a9270d-5034-4866-b8ad-d429b6e2b2c5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:56.968295 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a9270d-5034-4866-b8ad-d429b6e2b2c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:56.968573 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a9270d-5034-4866-b8ad-d429b6e2b2c5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:56.969693 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a9270d-5034-4866-b8ad-d429b6e2b2c5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:56.971144 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a9270d-5034-4866-b8ad-d429b6e2b2c5 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:56.972710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 27/56] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:45:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:45:56.979269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98eaf0a-5443-4bde-b162-280c55bd17fa None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:57.015619 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:57.081538 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98eaf0a-5443-4bde-b162-280c55bd17fa None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:57.113661 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21512fc-c913-4ab5-9f94-16854c06757a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:57.114270 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21512fc-c913-4ab5-9f94-16854c06757a None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:57.115059 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21512fc-c913-4ab5-9f94-16854c06757a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:57.115320 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21512fc-c913-4ab5-9f94-16854c06757a None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:57.115979 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21512fc-c913-4ab5-9f94-16854c06757a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:57.117538 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21512fc-c913-4ab5-9f94-16854c06757a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:57.198895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 30/57] 10.222.0.42 () {60 vars in 1140 bytes} [Sun Sep 4 00:45:56 2022] POST /identity/v3/projects => generated 298 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:45:59.170661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e92fc1d-4fd8-4fd3-8e78-0a2dedc90200 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:59.171205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e92fc1d-4fd8-4fd3-8e78-0a2dedc90200 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:59.171205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e92fc1d-4fd8-4fd3-8e78-0a2dedc90200 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:59.172554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 28/58] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:45:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:45:59.178399 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b87873-ac9c-4916-8796-bd933f097db9 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:59.178771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b87873-ac9c-4916-8796-bd933f097db9 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:59.178957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b87873-ac9c-4916-8796-bd933f097db9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:59.196808 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-b4b87873-ac9c-4916-8796-bd933f097db9 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:45:59.198027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b4b87873-ac9c-4916-8796-bd933f097db9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:45:59.263213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 31/59] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:45:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:45:59.545189 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e92fc1d-4fd8-4fd3-8e78-0a2dedc90200 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:59.587205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:59.646586 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e92fc1d-4fd8-4fd3-8e78-0a2dedc90200 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:59.679862 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349ec817-22fa-4c21-a6ce-5733e2d02b9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:59.680481 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349ec817-22fa-4c21-a6ce-5733e2d02b9a None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:59.680763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349ec817-22fa-4c21-a6ce-5733e2d02b9a None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:59.681037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349ec817-22fa-4c21-a6ce-5733e2d02b9a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:59.682175 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349ec817-22fa-4c21-a6ce-5733e2d02b9a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:59.683612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349ec817-22fa-4c21-a6ce-5733e2d02b9a None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:59.685177 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 29/60] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:45:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:45:59.690573 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b87873-ac9c-4916-8796-bd933f097db9 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:45:59.707414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:45:59.788451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b87873-ac9c-4916-8796-bd933f097db9 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:45:59.820899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1310dd6-a4ee-4901-8b2f-0c9cfcc3e654 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:45:59.821531 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1310dd6-a4ee-4901-8b2f-0c9cfcc3e654 None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:45:59.821813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1310dd6-a4ee-4901-8b2f-0c9cfcc3e654 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:45:59.822090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1310dd6-a4ee-4901-8b2f-0c9cfcc3e654 None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:45:59.822732 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1310dd6-a4ee-4901-8b2f-0c9cfcc3e654 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:45:59.824172 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1310dd6-a4ee-4901-8b2f-0c9cfcc3e654 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:45:59.924794 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 32/61] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:45:59 2022] POST /identity/v3/users => generated 275 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:46:02.958663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3fa8a8-fd69-4da3-b54d-cfef7a38cc86 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:02.959690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3fa8a8-fd69-4da3-b54d-cfef7a38cc86 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:02.960169 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3fa8a8-fd69-4da3-b54d-cfef7a38cc86 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:02.961727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 30/62] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:02.966880 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0621b5fb-b62f-48a8-b290-b9edc4d44173 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:02.967667 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0621b5fb-b62f-48a8-b290-b9edc4d44173 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:02.968225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0621b5fb-b62f-48a8-b290-b9edc4d44173 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:02.991062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-0621b5fb-b62f-48a8-b290-b9edc4d44173 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:02.992284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-0621b5fb-b62f-48a8-b290-b9edc4d44173 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:03.084089 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778ab5af-6111-4c30-88b6-db3ad0f74279 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:03.084639 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 33/63] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:03.085099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778ab5af-6111-4c30-88b6-db3ad0f74279 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:03.085508 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778ab5af-6111-4c30-88b6-db3ad0f74279 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:03.087085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 31/64] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:03 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:46:03.103636 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713f30b8-f525-4314-854a-6a107aebc594 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:03.103927 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713f30b8-f525-4314-854a-6a107aebc594 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:03.104206 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713f30b8-f525-4314-854a-6a107aebc594 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:03.117357 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:03.131653 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-713f30b8-f525-4314-854a-6a107aebc594 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:03.132871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-713f30b8-f525-4314-854a-6a107aebc594 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:03.226500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 34/65] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:03.232223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585a858e-6c24-484a-8bbb-d134f6e3bb8e None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:03.232223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585a858e-6c24-484a-8bbb-d134f6e3bb8e None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:03.232223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585a858e-6c24-484a-8bbb-d134f6e3bb8e None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:03.232223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 32/66] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:03.237680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd73d7f-32ff-4e52-b88f-4bd2357ab4c9 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:03.238152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd73d7f-32ff-4e52-b88f-4bd2357ab4c9 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:03.238592 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd73d7f-32ff-4e52-b88f-4bd2357ab4c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:03.258312 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-abd73d7f-32ff-4e52-b88f-4bd2357ab4c9 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:03.259727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-abd73d7f-32ff-4e52-b88f-4bd2357ab4c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:03.329292 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 35/67] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:03.447005 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585a858e-6c24-484a-8bbb-d134f6e3bb8e None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:03.538056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d3d887-7feb-4446-ac9b-eb1410020ca1 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:03.538638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d3d887-7feb-4446-ac9b-eb1410020ca1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:03.539102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d3d887-7feb-4446-ac9b-eb1410020ca1 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:03.540656 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 36/68] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:46:03.546218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de5033c-10e0-4a00-958a-e7c34d6956b7 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:03.546736 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de5033c-10e0-4a00-958a-e7c34d6956b7 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:03.547204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de5033c-10e0-4a00-958a-e7c34d6956b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:03.566235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-4de5033c-10e0-4a00-958a-e7c34d6956b7 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:03.568313 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-4de5033c-10e0-4a00-958a-e7c34d6956b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:03.593857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585a858e-6c24-484a-8bbb-d134f6e3bb8e None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:03.607701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:03.648003 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1076895-f02a-401c-8978-59b5f2695e0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:03.649287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1076895-f02a-401c-8978-59b5f2695e0d None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:03.649711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1076895-f02a-401c-8978-59b5f2695e0d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:03.650278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1076895-f02a-401c-8978-59b5f2695e0d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:03.655905 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 37/69] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:46:03.658036 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de5033c-10e0-4a00-958a-e7c34d6956b7 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:03.661094 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:03.675743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1076895-f02a-401c-8978-59b5f2695e0d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:03.682259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-d1076895-f02a-401c-8978-59b5f2695e0d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:03.682259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:03.682259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:03.682259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:03.682259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:03.682259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:03.682259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:03.682259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:03.682259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:03.682259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:03.682259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:03.682259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:03.682259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:03.682259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:03.682259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:03.682259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:03.683091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:03.683091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:03.683091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:03.683091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:03.683091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:03.683091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:03.683091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:03.683091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:03.683091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:03.683091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:03.683091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:03.683091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:03.683091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:03.683091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:03.683091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:03.683091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:03.683847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:03.683847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:03.683847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:03.683847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:03.683847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:03.683847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:03.683847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:03.683847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:03.683847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:03.683847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:03.683847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:03.683847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:03.690775 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 33/70] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:03 2022] GET /identity/v3/roles/member => generated 84 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:03.696978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1076895-f02a-401c-8978-59b5f2695e0d None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:03.805550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de5033c-10e0-4a00-958a-e7c34d6956b7 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:03.840370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1076895-f02a-401c-8978-59b5f2695e0d None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:03.841715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8d0938-62b2-43e3-97ea-a8d842ff7b74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:03.842359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8d0938-62b2-43e3-97ea-a8d842ff7b74 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:03.842645 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8d0938-62b2-43e3-97ea-a8d842ff7b74 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:03.842925 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8d0938-62b2-43e3-97ea-a8d842ff7b74 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:03.857376 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8d0938-62b2-43e3-97ea-a8d842ff7b74 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:03.866282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-6f8d0938-62b2-43e3-97ea-a8d842ff7b74 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:03.866282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:03.866282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:03.866282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:03.866282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:03.866282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:03.866282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:03.866282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:03.866282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:03.866282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:03.866282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:03.866282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:03.866282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:03.866282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:03.866282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:03.866282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:03.867115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:03.867115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:03.867115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:03.867115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:03.867115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:03.867115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:03.867115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:03.867115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:03.867115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:03.867115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:03.867115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:03.867115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:03.867115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:03.867115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:03.867115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:03.867115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:03.867844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:03.867844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:03.867844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:03.867844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:03.867844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:03.867844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:03.867844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:03.867844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:03.867844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:03.867844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:03.867844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:03.867844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:03.868412 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 38/71] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:46:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:03.869232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8d0938-62b2-43e3-97ea-a8d842ff7b74 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:03.877486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d7c99b-2397-4190-8223-4d7f17adeb80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:03.878104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d7c99b-2397-4190-8223-4d7f17adeb80 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:03.878813 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d7c99b-2397-4190-8223-4d7f17adeb80 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:03.879105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d7c99b-2397-4190-8223-4d7f17adeb80 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:03.879788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d7c99b-2397-4190-8223-4d7f17adeb80 None admin] RBAC: Adding query filter params (name=member) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:03.880679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d7c99b-2397-4190-8223-4d7f17adeb80 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:03.882469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d7c99b-2397-4190-8223-4d7f17adeb80 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:03.894290 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 34/72] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:03.899966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278024d3-e999-42b4-a1b7-3a87b7aa4231 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:03.900272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278024d3-e999-42b4-a1b7-3a87b7aa4231 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:03.900514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278024d3-e999-42b4-a1b7-3a87b7aa4231 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:03.901941 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 35/73] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:03 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:03.902509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278024d3-e999-42b4-a1b7-3a87b7aa4231 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:03.928385 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:03.939565 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:03.975804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8d0938-62b2-43e3-97ea-a8d842ff7b74 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:04.011161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ad8c93-4c8d-45d3-b202-909f0a4d0ec7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:04.011801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ad8c93-4c8d-45d3-b202-909f0a4d0ec7 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:04.012082 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ad8c93-4c8d-45d3-b202-909f0a4d0ec7 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:04.012357 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ad8c93-4c8d-45d3-b202-909f0a4d0ec7 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:04.015933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278024d3-e999-42b4-a1b7-3a87b7aa4231 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:04.022549 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ad8c93-4c8d-45d3-b202-909f0a4d0ec7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:04.028781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-00ad8c93-4c8d-45d3-b202-909f0a4d0ec7 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:04.028781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:04.028781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:04.028781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:04.028781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:04.028781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:04.028781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:04.028781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:04.028781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:04.028781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:04.028781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:04.028781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:04.028781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:04.028781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:04.028781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:04.028781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:04.029804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:04.029804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:04.029804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:04.029804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:04.029804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:04.029804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:04.029804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:04.029804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:04.029804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:04.029804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:04.029804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:04.029804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:04.029804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:04.029804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:04.029804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:04.029804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:04.030657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:04.030657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:04.030657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:04.030657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:04.030657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:04.030657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:04.030657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:04.030657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:04.030657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:04.030657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:04.030657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:04.030657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:04.030657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 39/74] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:46:03 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:04.030657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ad8c93-4c8d-45d3-b202-909f0a4d0ec7 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:04.052929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a3e5d8-befa-4854-9c44-f4ade6dff580 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:04.053629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a3e5d8-befa-4854-9c44-f4ade6dff580 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:04.053915 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a3e5d8-befa-4854-9c44-f4ade6dff580 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:04.054190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a3e5d8-befa-4854-9c44-f4ade6dff580 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:04.054858 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a3e5d8-befa-4854-9c44-f4ade6dff580 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:04.055164 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a3e5d8-befa-4854-9c44-f4ade6dff580 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:04.056629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a3e5d8-befa-4854-9c44-f4ade6dff580 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:04.064562 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 36/75] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:04.070421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5bce07-909d-4d55-8139-d6919f0639f9 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:04.070713 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5bce07-909d-4d55-8139-d6919f0639f9 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:04.070986 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5bce07-909d-4d55-8139-d6919f0639f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:04.088322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9f5bce07-909d-4d55-8139-d6919f0639f9 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:04.089861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9f5bce07-909d-4d55-8139-d6919f0639f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:04.133752 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ad8c93-4c8d-45d3-b202-909f0a4d0ec7 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:04.147149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:04.155767 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 37/76] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:04.156596 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5bce07-909d-4d55-8139-d6919f0639f9 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:04.167918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8943a1fa-be8a-4bbf-b088-e145c965109c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:04.168582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8943a1fa-be8a-4bbf-b088-e145c965109c None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:04.168876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8943a1fa-be8a-4bbf-b088-e145c965109c None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:04.169750 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8943a1fa-be8a-4bbf-b088-e145c965109c None admin] PATH_INFO: `/v3/users/d4a141c6519544a4ad26f923f1fb0382` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:04.185716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8943a1fa-be8a-4bbf-b088-e145c965109c None admin] RBAC: Authorizing `identity:get_user(user_id=d4a141c6519544a4ad26f923f1fb0382)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:04.187542 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8943a1fa-be8a-4bbf-b088-e145c965109c None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:04.189864 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:04.193908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 40/77] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:04 2022] GET /identity/v3/users/d4a141c6519544a4ad26f923f1fb0382 => generated 275 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:04.194580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8943a1fa-be8a-4bbf-b088-e145c965109c None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:04.281612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5bce07-909d-4d55-8139-d6919f0639f9 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:04.297006 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8943a1fa-be8a-4bbf-b088-e145c965109c None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:04.320728 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3338118-bcfe-4b3b-9365-30f852f0f6a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:04.321451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3338118-bcfe-4b3b-9365-30f852f0f6a0 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:04.321737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3338118-bcfe-4b3b-9365-30f852f0f6a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:04.322015 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3338118-bcfe-4b3b-9365-30f852f0f6a0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:04.330449 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da3e59b-6717-4953-9aaf-7bf3f862c0ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:04.331055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da3e59b-6717-4953-9aaf-7bf3f862c0ac None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:04.331338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da3e59b-6717-4953-9aaf-7bf3f862c0ac None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:04.331610 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da3e59b-6717-4953-9aaf-7bf3f862c0ac None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:04.332290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da3e59b-6717-4953-9aaf-7bf3f862c0ac None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:04.332599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da3e59b-6717-4953-9aaf-7bf3f862c0ac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:04.332748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3338118-bcfe-4b3b-9365-30f852f0f6a0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:04.334222 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da3e59b-6717-4953-9aaf-7bf3f862c0ac None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:04.338472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-b3338118-bcfe-4b3b-9365-30f852f0f6a0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:04.338472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:04.338472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:04.338472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:04.338472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:04.338472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:04.338472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:04.338472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:04.338472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:04.338472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:04.338472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:04.338472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:04.338472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:04.338472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:04.338472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:04.338472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:04.339276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:04.339276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:04.339276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:04.339276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:04.339276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:04.339276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:04.339276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:04.339276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:04.339276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:04.339276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:04.339276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:04.339276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:04.339276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:04.339276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:04.339276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:04.339276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:04.341512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:04.341512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:04.341512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:04.341512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:04.341512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:04.341512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:04.341512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:04.341512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:04.341512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:04.341512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:04.341512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:04.341512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:04.341512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 38/78] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:04 2022] GET /identity/v3/roles/member => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:04.341512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3338118-bcfe-4b3b-9365-30f852f0f6a0 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:04.347828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 41/79] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:46:04 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:04.349416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da3e59b-6717-4953-9aaf-7bf3f862c0ac None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:04.385856 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:04.440091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:04.459211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da3e59b-6717-4953-9aaf-7bf3f862c0ac None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:04.465477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3338118-bcfe-4b3b-9365-30f852f0f6a0 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:04.495528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805ca55a-9b08-4ecb-a32a-47cecb777da3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:04.496142 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805ca55a-9b08-4ecb-a32a-47cecb777da3 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:04.496418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805ca55a-9b08-4ecb-a32a-47cecb777da3 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:04.496933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805ca55a-9b08-4ecb-a32a-47cecb777da3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:04.500049 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040f2d30-5831-4c53-b2ec-7206cb529cd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:04.500677 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040f2d30-5831-4c53-b2ec-7206cb529cd9 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:04.500955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040f2d30-5831-4c53-b2ec-7206cb529cd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:04.501257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040f2d30-5831-4c53-b2ec-7206cb529cd9 None admin] PATH_INFO: `/v3/users/01782590dd294bd488a6d5c08a043ce1` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:04.502388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040f2d30-5831-4c53-b2ec-7206cb529cd9 None admin] RBAC: Authorizing `identity:get_user(user_id=01782590dd294bd488a6d5c08a043ce1)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:04.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040f2d30-5831-4c53-b2ec-7206cb529cd9 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:04.505495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 39/80] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:04 2022] GET /identity/v3/users/01782590dd294bd488a6d5c08a043ce1 => generated 247 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:04.506331 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040f2d30-5831-4c53-b2ec-7206cb529cd9 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:04.565274 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805ca55a-9b08-4ecb-a32a-47cecb777da3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:04.566738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805ca55a-9b08-4ecb-a32a-47cecb777da3 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:04.595852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:04.605873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040f2d30-5831-4c53-b2ec-7206cb529cd9 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:04.640863 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d69f9c-8cbe-4426-b1c6-d8f9db9fb6cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:04.641502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d69f9c-8cbe-4426-b1c6-d8f9db9fb6cf None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:04.641791 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d69f9c-8cbe-4426-b1c6-d8f9db9fb6cf None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:04.642070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d69f9c-8cbe-4426-b1c6-d8f9db9fb6cf None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:04.642738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d69f9c-8cbe-4426-b1c6-d8f9db9fb6cf None admin] RBAC: Adding query filter params (name=member) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:04.643045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d69f9c-8cbe-4426-b1c6-d8f9db9fb6cf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:04.644438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d69f9c-8cbe-4426-b1c6-d8f9db9fb6cf None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:04.653032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 40/81] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:04 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:04.653850 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d69f9c-8cbe-4426-b1c6-d8f9db9fb6cf None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:04.666348 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:04.667113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 42/82] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:04.668173 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805ca55a-9b08-4ecb-a32a-47cecb777da3 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:04.757443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d69f9c-8cbe-4426-b1c6-d8f9db9fb6cf None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:04.769027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805ca55a-9b08-4ecb-a32a-47cecb777da3 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:04.791795 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac95b9c-399d-426c-8104-781ffc5486af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:04.792431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac95b9c-399d-426c-8104-781ffc5486af None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:04.792717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac95b9c-399d-426c-8104-781ffc5486af None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:04.792993 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac95b9c-399d-426c-8104-781ffc5486af None admin] PATH_INFO: `/v3/users/d4a141c6519544a4ad26f923f1fb0382` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:04.794178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac95b9c-399d-426c-8104-781ffc5486af None admin] RBAC: Authorizing `identity:get_user(user_id=d4a141c6519544a4ad26f923f1fb0382)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:04.795621 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac95b9c-399d-426c-8104-781ffc5486af None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:04.797222 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 41/83] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:04 2022] GET /identity/v3/users/d4a141c6519544a4ad26f923f1fb0382 => generated 275 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:04.797766 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:04.797984 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac95b9c-399d-426c-8104-781ffc5486af None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:04.803631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a901d781-0809-41f1-82d5-ba247e8abe10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:04.804273 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a901d781-0809-41f1-82d5-ba247e8abe10 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:04.804552 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a901d781-0809-41f1-82d5-ba247e8abe10 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:04.804827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a901d781-0809-41f1-82d5-ba247e8abe10 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:04.805963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a901d781-0809-41f1-82d5-ba247e8abe10 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:04.807402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a901d781-0809-41f1-82d5-ba247e8abe10 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:04.808955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 43/84] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:46:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:04.808955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a901d781-0809-41f1-82d5-ba247e8abe10 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:04.878408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:04.901040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac95b9c-399d-426c-8104-781ffc5486af None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:04.908853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a901d781-0809-41f1-82d5-ba247e8abe10 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:04.934074 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4171186d-3cc3-4149-a836-bb9a7a2fa1d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:04.934681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4171186d-3cc3-4149-a836-bb9a7a2fa1d1 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:04.934963 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4171186d-3cc3-4149-a836-bb9a7a2fa1d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:04.935239 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4171186d-3cc3-4149-a836-bb9a7a2fa1d1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:04.942049 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964de753-1200-48e7-ae43-15e172e5d159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:04.943238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964de753-1200-48e7-ae43-15e172e5d159 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:04.943518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964de753-1200-48e7-ae43-15e172e5d159 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:04.943793 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964de753-1200-48e7-ae43-15e172e5d159 None admin] PATH_INFO: `/v3/users/d4a141c6519544a4ad26f923f1fb0382` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:04.945367 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964de753-1200-48e7-ae43-15e172e5d159 None admin] RBAC: Authorizing `identity:get_user(user_id=d4a141c6519544a4ad26f923f1fb0382)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:04.947167 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964de753-1200-48e7-ae43-15e172e5d159 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:04.949099 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 44/85] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:04 2022] GET /identity/v3/users/d4a141c6519544a4ad26f923f1fb0382 => generated 275 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:04.949865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964de753-1200-48e7-ae43-15e172e5d159 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:05.003205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4171186d-3cc3-4149-a836-bb9a7a2fa1d1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:05.004690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4171186d-3cc3-4149-a836-bb9a7a2fa1d1 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:05.018823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:05.051535 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964de753-1200-48e7-ae43-15e172e5d159 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:05.083613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:05.086537 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6ba4ff-2cf6-4a7e-a25b-df85af860590 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:05.087169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6ba4ff-2cf6-4a7e-a25b-df85af860590 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:05.087454 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6ba4ff-2cf6-4a7e-a25b-df85af860590 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:05.087730 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6ba4ff-2cf6-4a7e-a25b-df85af860590 None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:05.088857 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6ba4ff-2cf6-4a7e-a25b-df85af860590 None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:05.090333 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6ba4ff-2cf6-4a7e-a25b-df85af860590 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:05.091961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 45/86] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:04 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:05.095319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6ba4ff-2cf6-4a7e-a25b-df85af860590 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:05.108135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 42/87] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:05.108961 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4171186d-3cc3-4149-a836-bb9a7a2fa1d1 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:05.194719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6ba4ff-2cf6-4a7e-a25b-df85af860590 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:05.207106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4171186d-3cc3-4149-a836-bb9a7a2fa1d1 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:05.227765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a597a0-186f-45a9-952e-e913fc6aea07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:05.228376 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a597a0-186f-45a9-952e-e913fc6aea07 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:05.228659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a597a0-186f-45a9-952e-e913fc6aea07 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:05.228934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a597a0-186f-45a9-952e-e913fc6aea07 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:05.232328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:05.241164 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642771e0-3a5b-41b1-99f0-79b15471b434 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:05.241775 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642771e0-3a5b-41b1-99f0-79b15471b434 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:05.242058 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642771e0-3a5b-41b1-99f0-79b15471b434 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:05.242334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642771e0-3a5b-41b1-99f0-79b15471b434 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:05.242984 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642771e0-3a5b-41b1-99f0-79b15471b434 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:05.244501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642771e0-3a5b-41b1-99f0-79b15471b434 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:05.295959 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a597a0-186f-45a9-952e-e913fc6aea07 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:05.297473 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a597a0-186f-45a9-952e-e913fc6aea07 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:05.340798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 43/88] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:46:05 2022] POST /identity/v3/users => generated 289 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:46:05.341784 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642771e0-3a5b-41b1-99f0-79b15471b434 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:05.390312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:05.405707 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 46/89] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:05.407019 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a597a0-186f-45a9-952e-e913fc6aea07 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:05.455469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:05.475652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642771e0-3a5b-41b1-99f0-79b15471b434 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:05.522897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bc2234-04a2-4649-b504-b7d170d7a223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:05.522897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bc2234-04a2-4649-b504-b7d170d7a223 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:05.522897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bc2234-04a2-4649-b504-b7d170d7a223 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:05.522897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bc2234-04a2-4649-b504-b7d170d7a223 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:05.532603 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a597a0-186f-45a9-952e-e913fc6aea07 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:05.575681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83ae955-7161-4478-bb90-345cb5799f8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:05.577437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83ae955-7161-4478-bb90-345cb5799f8e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:05.577437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83ae955-7161-4478-bb90-345cb5799f8e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:05.577437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83ae955-7161-4478-bb90-345cb5799f8e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:05.577910 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83ae955-7161-4478-bb90-345cb5799f8e None admin] RBAC: Adding query filter params (role.id=4bfd6d0544ee4a9e933538e78135a12c, scope.project.id=1193eb16a76d432faf6141e47cc99c5a, user.id=d4a141c6519544a4ad26f923f1fb0382) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:05.578396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83ae955-7161-4478-bb90-345cb5799f8e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:05.580401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83ae955-7161-4478-bb90-345cb5799f8e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:05.587489 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 47/90] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:05 2022] GET /identity/v3/role_assignments?user.id=d4a141c6519544a4ad26f923f1fb0382&scope.project.id=1193eb16a76d432faf6141e47cc99c5a&role.id=4bfd6d0544ee4a9e933538e78135a12c => generated 259 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:05.588575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83ae955-7161-4478-bb90-345cb5799f8e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:05.605282 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bc2234-04a2-4649-b504-b7d170d7a223 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:05.606803 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bc2234-04a2-4649-b504-b7d170d7a223 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:05.659380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:05.731037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:05.734516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83ae955-7161-4478-bb90-345cb5799f8e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:05.744035 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 44/91] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:05.744937 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bc2234-04a2-4649-b504-b7d170d7a223 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:05.782125 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffd8d08-355a-4537-bf09-0c164393e14d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:05.782785 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffd8d08-355a-4537-bf09-0c164393e14d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:05.783065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffd8d08-355a-4537-bf09-0c164393e14d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:05.783338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffd8d08-355a-4537-bf09-0c164393e14d None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:05.784512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffd8d08-355a-4537-bf09-0c164393e14d None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:05.786042 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffd8d08-355a-4537-bf09-0c164393e14d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:05.787767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 48/92] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:05 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:05.788561 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffd8d08-355a-4537-bf09-0c164393e14d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:05.890468 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bc2234-04a2-4649-b504-b7d170d7a223 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:05.903760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffd8d08-355a-4537-bf09-0c164393e14d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:05.940394 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a530cfb2-97df-4292-87f7-1d528fb89598 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:05.942188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a530cfb2-97df-4292-87f7-1d528fb89598 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:05.948526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a530cfb2-97df-4292-87f7-1d528fb89598 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:05.948526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a530cfb2-97df-4292-87f7-1d528fb89598 None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:05.948526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a530cfb2-97df-4292-87f7-1d528fb89598 None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:05.948526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a530cfb2-97df-4292-87f7-1d528fb89598 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:05.948526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 45/93] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:05 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:05.949058 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a530cfb2-97df-4292-87f7-1d528fb89598 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:05.954181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c1fdcf-07b4-434c-b8e7-4012723fe2a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:05.954831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c1fdcf-07b4-434c-b8e7-4012723fe2a9 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:05.955111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c1fdcf-07b4-434c-b8e7-4012723fe2a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:05.955382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c1fdcf-07b4-434c-b8e7-4012723fe2a9 None admin] PATH_INFO: `/v3/projects/3cd6beb90b3c45c5bb9adeac3df910e1` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:05.956516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c1fdcf-07b4-434c-b8e7-4012723fe2a9 None admin] RBAC: Authorizing `identity:get_project(project_id=3cd6beb90b3c45c5bb9adeac3df910e1)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:05.958022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c1fdcf-07b4-434c-b8e7-4012723fe2a9 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:05.967444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:05.967851 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 49/94] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:05 2022] GET /identity/v3/projects/3cd6beb90b3c45c5bb9adeac3df910e1 => generated 312 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:05.968766 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c1fdcf-07b4-434c-b8e7-4012723fe2a9 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:06.054612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:06.101179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a530cfb2-97df-4292-87f7-1d528fb89598 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:06.155642 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194ec303-d577-41fc-9ade-6a10d921650a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:06.156260 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194ec303-d577-41fc-9ade-6a10d921650a None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:06.156543 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194ec303-d577-41fc-9ade-6a10d921650a None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:06.156822 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194ec303-d577-41fc-9ade-6a10d921650a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:06.157595 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194ec303-d577-41fc-9ade-6a10d921650a None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=1193eb16a76d432faf6141e47cc99c5a, user.id=01782590dd294bd488a6d5c08a043ce1) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:06.157908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194ec303-d577-41fc-9ade-6a10d921650a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:06.159379 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194ec303-d577-41fc-9ade-6a10d921650a None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:06.173254 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 46/95] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:05 2022] GET /identity/v3/role_assignments?user.id=01782590dd294bd488a6d5c08a043ce1&scope.project.id=1193eb16a76d432faf6141e47cc99c5a&role.id=19c006412bec45dc8febd7cc86b9678f => generated 259 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:06.174098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194ec303-d577-41fc-9ade-6a10d921650a None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:06.193994 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c1fdcf-07b4-434c-b8e7-4012723fe2a9 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:06.234561 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5df39d4-d3f7-43e2-b367-4282a70aef50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:06.235223 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5df39d4-d3f7-43e2-b367-4282a70aef50 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:06.235449 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5df39d4-d3f7-43e2-b367-4282a70aef50 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:06.235726 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5df39d4-d3f7-43e2-b367-4282a70aef50 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:06.236490 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5df39d4-d3f7-43e2-b367-4282a70aef50 None admin] RBAC: Adding query filter params (role.id=c599b217579e42a49c657c642ea0648f, scope.project.id=1193eb16a76d432faf6141e47cc99c5a, user.id=d4a141c6519544a4ad26f923f1fb0382) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:06.236803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5df39d4-d3f7-43e2-b367-4282a70aef50 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:06.241099 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5df39d4-d3f7-43e2-b367-4282a70aef50 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:06.244809 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 50/96] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:05 2022] GET /identity/v3/role_assignments?user.id=d4a141c6519544a4ad26f923f1fb0382&scope.project.id=1193eb16a76d432faf6141e47cc99c5a&role.id=c599b217579e42a49c657c642ea0648f => generated 259 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:06.303327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:06.365216 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194ec303-d577-41fc-9ade-6a10d921650a None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:06.413921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc795ad-243c-4116-88fd-c5bd2611355b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:06.414552 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc795ad-243c-4116-88fd-c5bd2611355b None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:06.414832 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc795ad-243c-4116-88fd-c5bd2611355b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:06.415103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc795ad-243c-4116-88fd-c5bd2611355b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:06.415852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc795ad-243c-4116-88fd-c5bd2611355b None admin] RBAC: Adding query filter params (role.id=4bfd6d0544ee4a9e933538e78135a12c, scope.project.id=3cd6beb90b3c45c5bb9adeac3df910e1, user.id=d4a141c6519544a4ad26f923f1fb0382) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:06.416158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc795ad-243c-4116-88fd-c5bd2611355b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:06.417777 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc795ad-243c-4116-88fd-c5bd2611355b None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:06.434510 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 47/97] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:06 2022] GET /identity/v3/role_assignments?user.id=d4a141c6519544a4ad26f923f1fb0382&scope.project.id=3cd6beb90b3c45c5bb9adeac3df910e1&role.id=4bfd6d0544ee4a9e933538e78135a12c => generated 259 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:07.980204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f7857b-ae28-49ee-8981-4eaf3d173f13 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:07.981252 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f7857b-ae28-49ee-8981-4eaf3d173f13 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:07.981745 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f7857b-ae28-49ee-8981-4eaf3d173f13 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:07.983402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 51/98] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:07.994628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcb9856-bcfc-4195-beda-953585d676f7 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:07.995500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcb9856-bcfc-4195-beda-953585d676f7 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:07.995981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcb9856-bcfc-4195-beda-953585d676f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:08.020861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-fbcb9856-bcfc-4195-beda-953585d676f7 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:08.022423 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-fbcb9856-bcfc-4195-beda-953585d676f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:08.105598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 48/99] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:08.154335 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cd7b29-a976-4209-bee2-f5c98314565d None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:08.154713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cd7b29-a976-4209-bee2-f5c98314565d None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:08.155067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cd7b29-a976-4209-bee2-f5c98314565d None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:08.156492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 52/100] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:08.162243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d04bf2a-2ba3-4e1f-a407-36ec3fce41ef None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:08.162521 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d04bf2a-2ba3-4e1f-a407-36ec3fce41ef None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:08.162794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d04bf2a-2ba3-4e1f-a407-36ec3fce41ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:08.190020 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9d04bf2a-2ba3-4e1f-a407-36ec3fce41ef None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:08.191214 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9d04bf2a-2ba3-4e1f-a407-36ec3fce41ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:08.263875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:08.274603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 49/101] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:08.578921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418da696-60b9-4831-868b-dff65bcf3cc7 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:08.579205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418da696-60b9-4831-868b-dff65bcf3cc7 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:08.579912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418da696-60b9-4831-868b-dff65bcf3cc7 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:08.581537 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 53/102] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:08.587922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea85282-b923-490d-9608-a73e1662715a None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:08.588211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea85282-b923-490d-9608-a73e1662715a None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:08.588498 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea85282-b923-490d-9608-a73e1662715a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:08.617347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-3ea85282-b923-490d-9608-a73e1662715a None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:08.618539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-3ea85282-b923-490d-9608-a73e1662715a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:08.699461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418da696-60b9-4831-868b-dff65bcf3cc7 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:08.712631 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 50/103] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:08.830021 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:08.833913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418da696-60b9-4831-868b-dff65bcf3cc7 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:08.869022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea85282-b923-490d-9608-a73e1662715a None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:08.877058 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47c0aac-688b-4c51-86a0-d3f1c35f843d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:08.877723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47c0aac-688b-4c51-86a0-d3f1c35f843d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:08.878007 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47c0aac-688b-4c51-86a0-d3f1c35f843d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:08.878289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47c0aac-688b-4c51-86a0-d3f1c35f843d None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:08.896433 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47c0aac-688b-4c51-86a0-d3f1c35f843d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:08.902115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-f47c0aac-688b-4c51-86a0-d3f1c35f843d None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:08.902115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:08.902115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:08.902115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:08.902115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:08.902115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:08.902115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:08.902115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:08.902115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:08.902115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:08.902115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:08.902115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:08.902115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:08.902115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:08.902115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:08.902115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:08.902948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:08.902948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:08.902948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:08.902948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:08.902948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:08.902948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:08.902948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:08.902948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:08.902948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:08.902948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:08.902948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:08.902948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:08.902948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:08.902948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:08.902948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:08.902948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:08.903721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:08.903721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:08.903721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:08.903721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:08.903721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:08.903721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:08.903721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:08.903721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:08.903721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:08.903721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:08.903721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:08.903721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:08.903721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 54/104] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:08 2022] GET /identity/v3/roles/reader => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:46:08.912854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47c0aac-688b-4c51-86a0-d3f1c35f843d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:09.036117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea85282-b923-490d-9608-a73e1662715a None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:09.048085 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47c0aac-688b-4c51-86a0-d3f1c35f843d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:09.065430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:09.089875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ef7eee-9e7c-4a5a-bbef-f189ccb7911a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:09.090827 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ef7eee-9e7c-4a5a-bbef-f189ccb7911a None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:09.091318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ef7eee-9e7c-4a5a-bbef-f189ccb7911a None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:09.091780 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ef7eee-9e7c-4a5a-bbef-f189ccb7911a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:09.094535 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ef7eee-9e7c-4a5a-bbef-f189ccb7911a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:09.101741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ef7eee-9e7c-4a5a-bbef-f189ccb7911a None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:09.103819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 51/105] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:46:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:09.105106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d1631f-2bbe-4077-aa15-0e1a2afe50c7 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:09.105429 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d1631f-2bbe-4077-aa15-0e1a2afe50c7 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:09.105693 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d1631f-2bbe-4077-aa15-0e1a2afe50c7 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:09.107051 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 52/106] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:09.111693 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35283606-e7ea-41e9-bd11-03bbfd43d046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:09.112466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d63cdb2-9d4a-42d0-bdc2-ce54fe413a3a None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:09.112758 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d63cdb2-9d4a-42d0-bdc2-ce54fe413a3a None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:09.112944 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35283606-e7ea-41e9-bd11-03bbfd43d046 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:09.113038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d63cdb2-9d4a-42d0-bdc2-ce54fe413a3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:09.113540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35283606-e7ea-41e9-bd11-03bbfd43d046 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:09.114000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35283606-e7ea-41e9-bd11-03bbfd43d046 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:09.115453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35283606-e7ea-41e9-bd11-03bbfd43d046 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:09.115949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35283606-e7ea-41e9-bd11-03bbfd43d046 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:09.117643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35283606-e7ea-41e9-bd11-03bbfd43d046 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:09.127291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 55/107] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:08 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:09.128352 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35283606-e7ea-41e9-bd11-03bbfd43d046 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:09.143113 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-2d63cdb2-9d4a-42d0-bdc2-ce54fe413a3a None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:09.144220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-2d63cdb2-9d4a-42d0-bdc2-ce54fe413a3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:09.159943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:09.218593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 53/108] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:09.219523 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d63cdb2-9d4a-42d0-bdc2-ce54fe413a3a None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:09.244857 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35283606-e7ea-41e9-bd11-03bbfd43d046 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:09.281749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92373aee-e4c2-4ff5-ae70-b2819c96949c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:09.282362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92373aee-e4c2-4ff5-ae70-b2819c96949c None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:09.282644 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92373aee-e4c2-4ff5-ae70-b2819c96949c None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:09.282919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92373aee-e4c2-4ff5-ae70-b2819c96949c None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:09.283571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92373aee-e4c2-4ff5-ae70-b2819c96949c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:09.285147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92373aee-e4c2-4ff5-ae70-b2819c96949c None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:09.338769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d63cdb2-9d4a-42d0-bdc2-ce54fe413a3a None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:09.358439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:09.376013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1244257-2fbd-401f-8db2-6aa00982da86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:09.376875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1244257-2fbd-401f-8db2-6aa00982da86 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:09.377286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1244257-2fbd-401f-8db2-6aa00982da86 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:09.377581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1244257-2fbd-401f-8db2-6aa00982da86 None admin] PATH_INFO: `/v3/users/381b26af45ac4d8d967cfbe0d8639173` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:09.381204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 56/109] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 4 00:46:09 2022] POST /identity/v3/projects => generated 302 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:46:09.383046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d2463d-a922-4604-9bf3-0069112d1bac None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:09.383584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d2463d-a922-4604-9bf3-0069112d1bac None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:09.384081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d2463d-a922-4604-9bf3-0069112d1bac None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:09.385754 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 57/110] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:09.387639 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d2463d-a922-4604-9bf3-0069112d1bac None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:09.393485 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1244257-2fbd-401f-8db2-6aa00982da86 None admin] RBAC: Authorizing `identity:get_user(user_id=381b26af45ac4d8d967cfbe0d8639173)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:09.396082 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1244257-2fbd-401f-8db2-6aa00982da86 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:09.401331 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 54/111] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:09 2022] GET /identity/v3/users/381b26af45ac4d8d967cfbe0d8639173 => generated 289 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:09.401331 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49aaaab5-1985-40c9-9c2c-99273c13fdf0 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:09.401331 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49aaaab5-1985-40c9-9c2c-99273c13fdf0 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:09.401331 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49aaaab5-1985-40c9-9c2c-99273c13fdf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:09.421174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-49aaaab5-1985-40c9-9c2c-99273c13fdf0 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:09.422514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-49aaaab5-1985-40c9-9c2c-99273c13fdf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:09.557650 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 55/112] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:09.559002 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49aaaab5-1985-40c9-9c2c-99273c13fdf0 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:09.572324 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:09.607238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d2463d-a922-4604-9bf3-0069112d1bac None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:09.646261 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a510cfb-a1ca-4983-a36f-1a99eedf1e46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:09.647277 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a510cfb-a1ca-4983-a36f-1a99eedf1e46 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:09.647568 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a510cfb-a1ca-4983-a36f-1a99eedf1e46 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:09.647977 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a510cfb-a1ca-4983-a36f-1a99eedf1e46 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:09.658654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a510cfb-a1ca-4983-a36f-1a99eedf1e46 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:09.664713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-9a510cfb-a1ca-4983-a36f-1a99eedf1e46 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:09.664713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:09.664713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:09.664713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:09.664713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:09.664713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:09.664713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:09.664713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:09.664713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:09.664713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:09.664713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:09.664713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:09.664713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:09.664713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:09.664713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:09.664713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:09.665779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:09.665779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:09.665779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:09.665779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:09.665779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:09.665779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:09.665779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:09.665779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:09.665779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:09.665779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:09.665779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:09.665779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:09.665779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:09.665779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:09.665779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:09.665779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:09.666727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:09.666727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:09.666727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:09.666727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:09.666727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:09.666727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:09.666727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:09.666727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:09.666727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:09.666727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:09.666727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:09.666727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:09.669103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 58/113] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:09.670040 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a510cfb-a1ca-4983-a36f-1a99eedf1e46 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:09.686280 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49aaaab5-1985-40c9-9c2c-99273c13fdf0 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:09.697281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:09.729828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01405f7-2799-4bf0-919b-c9e4a3b6085b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:09.730429 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01405f7-2799-4bf0-919b-c9e4a3b6085b None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:09.730710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01405f7-2799-4bf0-919b-c9e4a3b6085b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:09.730981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01405f7-2799-4bf0-919b-c9e4a3b6085b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:09.798656 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a510cfb-a1ca-4983-a36f-1a99eedf1e46 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:09.803436 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01405f7-2799-4bf0-919b-c9e4a3b6085b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:09.804387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01405f7-2799-4bf0-919b-c9e4a3b6085b None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:09.843591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096e73d8-8220-4441-aabb-dae654fd251d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:09.843863 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096e73d8-8220-4441-aabb-dae654fd251d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:09.844254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096e73d8-8220-4441-aabb-dae654fd251d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:09.844616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096e73d8-8220-4441-aabb-dae654fd251d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:09.846664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:09.856474 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096e73d8-8220-4441-aabb-dae654fd251d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:09.861359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-096e73d8-8220-4441-aabb-dae654fd251d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:09.861359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:09.861359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:09.861359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:09.861359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:09.861359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:09.861359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:09.861359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:09.861359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:09.861359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:09.861359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:09.861359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:09.861359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:09.861359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:09.861359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:09.861359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:09.868288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:09.868288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:09.868288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:09.868288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:09.868288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:09.868288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:09.868288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:09.868288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:09.868288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:09.868288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:09.868288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:09.868288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:09.868288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:09.868288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:09.868288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:09.868288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:09.869048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:09.869048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:09.869048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:09.869048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:09.869048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:09.869048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:09.869048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:09.869048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:09.869048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:09.869048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:09.869048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:09.869048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:09.869048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 59/114] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:46:09 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:09.869048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1d2c3f-d8d3-4fe7-abf1-db703c688f42 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:09.869740 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1d2c3f-d8d3-4fe7-abf1-db703c688f42 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:09.869740 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1d2c3f-d8d3-4fe7-abf1-db703c688f42 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:09.869740 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 60/115] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:09.869740 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1d2c3f-d8d3-4fe7-abf1-db703c688f42 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:09.923070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 56/116] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:09.928931 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01405f7-2799-4bf0-919b-c9e4a3b6085b None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:09.937629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:09.980300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1d2c3f-d8d3-4fe7-abf1-db703c688f42 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:10.016354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93dadb1-2522-4961-953e-a70f2bbd98a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:10.016963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93dadb1-2522-4961-953e-a70f2bbd98a4 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:10.017272 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93dadb1-2522-4961-953e-a70f2bbd98a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:10.017544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93dadb1-2522-4961-953e-a70f2bbd98a4 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:10.018260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93dadb1-2522-4961-953e-a70f2bbd98a4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:10.018610 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93dadb1-2522-4961-953e-a70f2bbd98a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:10.020004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93dadb1-2522-4961-953e-a70f2bbd98a4 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:10.029245 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 61/117] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:10.030250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93dadb1-2522-4961-953e-a70f2bbd98a4 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:10.049409 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01405f7-2799-4bf0-919b-c9e4a3b6085b None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:10.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d073cd73-f9a8-4e9e-ad42-6f3f523c4573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:10.088120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d073cd73-f9a8-4e9e-ad42-6f3f523c4573 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:10.088403 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d073cd73-f9a8-4e9e-ad42-6f3f523c4573 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:10.088678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d073cd73-f9a8-4e9e-ad42-6f3f523c4573 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:10.093300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:10.100323 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d073cd73-f9a8-4e9e-ad42-6f3f523c4573 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:10.105263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-d073cd73-f9a8-4e9e-ad42-6f3f523c4573 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:10.105263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:10.105263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:10.105263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:10.105263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:10.105263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:10.105263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:10.105263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:10.105263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:10.105263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:10.105263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:10.105263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:10.105263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:10.105263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:10.105263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:10.105263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:10.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:10.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:10.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:10.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:10.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:10.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:10.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:10.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:10.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:10.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:10.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:10.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:10.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:10.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:10.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:10.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:10.110249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:10.110249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:10.110249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:10.110249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:10.110249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:10.110249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:10.110249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:10.110249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:10.110249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:10.110249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:10.110249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:10.110249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:10.110249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 57/118] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:46:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:10.110249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c019f432-1a22-4de1-8cff-af8514b7f5b1 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:10.110999 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c019f432-1a22-4de1-8cff-af8514b7f5b1 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:10.110999 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c019f432-1a22-4de1-8cff-af8514b7f5b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:10.126580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-c019f432-1a22-4de1-8cff-af8514b7f5b1 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:10.127768 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c019f432-1a22-4de1-8cff-af8514b7f5b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:10.146581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93dadb1-2522-4961-953e-a70f2bbd98a4 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:10.181782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2c0197-eaf3-489b-93a6-356bf2da03f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:10.182371 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2c0197-eaf3-489b-93a6-356bf2da03f4 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:10.182593 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2c0197-eaf3-489b-93a6-356bf2da03f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:10.182862 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2c0197-eaf3-489b-93a6-356bf2da03f4 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:10.183559 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2c0197-eaf3-489b-93a6-356bf2da03f4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:10.183866 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2c0197-eaf3-489b-93a6-356bf2da03f4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:10.185247 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2c0197-eaf3-489b-93a6-356bf2da03f4 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:10.193649 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 62/119] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:46:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:10.194611 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:10.198526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2c0197-eaf3-489b-93a6-356bf2da03f4 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:10.211196 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 58/120] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:10.221171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c019f432-1a22-4de1-8cff-af8514b7f5b1 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:10.333540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2c0197-eaf3-489b-93a6-356bf2da03f4 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:10.350156 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c019f432-1a22-4de1-8cff-af8514b7f5b1 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:10.356806 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:10.370814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40861c31-fb31-4bbe-b6db-640d4e183107 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:10.371703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40861c31-fb31-4bbe-b6db-640d4e183107 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:10.372385 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40861c31-fb31-4bbe-b6db-640d4e183107 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:10.372805 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40861c31-fb31-4bbe-b6db-640d4e183107 None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:10.374313 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40861c31-fb31-4bbe-b6db-640d4e183107 None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:10.375904 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40861c31-fb31-4bbe-b6db-640d4e183107 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:10.383385 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 63/121] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:10 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:10.384763 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40861c31-fb31-4bbe-b6db-640d4e183107 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:10.387095 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6415b598-aebc-44e4-a7d1-fea78e129701 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:10.387751 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6415b598-aebc-44e4-a7d1-fea78e129701 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:10.388024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6415b598-aebc-44e4-a7d1-fea78e129701 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:10.388296 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6415b598-aebc-44e4-a7d1-fea78e129701 None admin] PATH_INFO: `/v3/users/d4a141c6519544a4ad26f923f1fb0382` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:10.389462 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6415b598-aebc-44e4-a7d1-fea78e129701 None admin] RBAC: Authorizing `identity:get_user(user_id=d4a141c6519544a4ad26f923f1fb0382)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:10.390961 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6415b598-aebc-44e4-a7d1-fea78e129701 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:10.392549 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 59/122] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:10 2022] GET /identity/v3/users/d4a141c6519544a4ad26f923f1fb0382 => generated 275 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:10.397885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6415b598-aebc-44e4-a7d1-fea78e129701 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:10.464020 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:10.503996 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6415b598-aebc-44e4-a7d1-fea78e129701 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:10.504415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40861c31-fb31-4bbe-b6db-640d4e183107 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:10.540605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289ecdf1-b880-45c8-87c7-8de31313be60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:10.541259 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289ecdf1-b880-45c8-87c7-8de31313be60 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:10.541576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289ecdf1-b880-45c8-87c7-8de31313be60 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:10.541817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289ecdf1-b880-45c8-87c7-8de31313be60 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:10.542186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bb4362-c679-462c-ba7e-1af6dfc34a1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:10.542497 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289ecdf1-b880-45c8-87c7-8de31313be60 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:10.542805 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289ecdf1-b880-45c8-87c7-8de31313be60 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:10.542894 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bb4362-c679-462c-ba7e-1af6dfc34a1c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:10.543093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bb4362-c679-462c-ba7e-1af6dfc34a1c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:10.543369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44bb4362-c679-462c-ba7e-1af6dfc34a1c None admin] PATH_INFO: `/v3/users/d4a141c6519544a4ad26f923f1fb0382` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:10.544171 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289ecdf1-b880-45c8-87c7-8de31313be60 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:10.544498 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bb4362-c679-462c-ba7e-1af6dfc34a1c None admin] RBAC: Authorizing `identity:get_user(user_id=d4a141c6519544a4ad26f923f1fb0382)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:10.545948 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44bb4362-c679-462c-ba7e-1af6dfc34a1c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:10.547717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 60/123] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:10 2022] GET /identity/v3/users/d4a141c6519544a4ad26f923f1fb0382 => generated 275 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:10.548408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bb4362-c679-462c-ba7e-1af6dfc34a1c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:10.557549 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 64/124] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:10.558423 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289ecdf1-b880-45c8-87c7-8de31313be60 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:10.614016 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:10.655099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44bb4362-c679-462c-ba7e-1af6dfc34a1c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:10.670730 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289ecdf1-b880-45c8-87c7-8de31313be60 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:10.688196 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:10.693120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50997ef-c915-4f94-8783-1bd821f9d170 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:10.699253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50997ef-c915-4f94-8783-1bd821f9d170 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:10.699253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50997ef-c915-4f94-8783-1bd821f9d170 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:10.699253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50997ef-c915-4f94-8783-1bd821f9d170 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:10.699253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50997ef-c915-4f94-8783-1bd821f9d170 None admin] RBAC: Adding query filter params (role.id=6b0968db45c64a66b001a39d40579011, scope.project.id=1193eb16a76d432faf6141e47cc99c5a, user.id=381b26af45ac4d8d967cfbe0d8639173) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:10.699253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50997ef-c915-4f94-8783-1bd821f9d170 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:10.699253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50997ef-c915-4f94-8783-1bd821f9d170 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:10.701976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 61/125] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:10 2022] GET /identity/v3/role_assignments?user.id=381b26af45ac4d8d967cfbe0d8639173&scope.project.id=1193eb16a76d432faf6141e47cc99c5a&role.id=6b0968db45c64a66b001a39d40579011 => generated 259 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:10.702755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50997ef-c915-4f94-8783-1bd821f9d170 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:10.720394 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497473c6-a07f-426d-917a-20df4333912e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:10.721889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497473c6-a07f-426d-917a-20df4333912e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:10.722436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497473c6-a07f-426d-917a-20df4333912e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:10.723057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497473c6-a07f-426d-917a-20df4333912e None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:10.725653 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497473c6-a07f-426d-917a-20df4333912e None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:10.728040 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497473c6-a07f-426d-917a-20df4333912e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:10.730684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 65/126] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:10 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:10.731879 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497473c6-a07f-426d-917a-20df4333912e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:10.851786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50997ef-c915-4f94-8783-1bd821f9d170 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:10.853197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497473c6-a07f-426d-917a-20df4333912e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:10.870350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:10.889344 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86353d00-13e4-4a84-90c2-d1a1385132b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:10.890002 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86353d00-13e4-4a84-90c2-d1a1385132b8 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:10.890301 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86353d00-13e4-4a84-90c2-d1a1385132b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:10.890561 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86353d00-13e4-4a84-90c2-d1a1385132b8 None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:10.891689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86353d00-13e4-4a84-90c2-d1a1385132b8 None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:10.893262 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86353d00-13e4-4a84-90c2-d1a1385132b8 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:10.895048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 66/127] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:10 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:10.896327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86353d00-13e4-4a84-90c2-d1a1385132b8 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:10.901752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bce3596-5670-43ca-aa68-61bf5dd4e7ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:10.902659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bce3596-5670-43ca-aa68-61bf5dd4e7ea None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:10.903139 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bce3596-5670-43ca-aa68-61bf5dd4e7ea None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:10.903628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bce3596-5670-43ca-aa68-61bf5dd4e7ea None admin] PATH_INFO: `/v3/roles/member` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:10.918269 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bce3596-5670-43ca-aa68-61bf5dd4e7ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:10.926574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-4bce3596-5670-43ca-aa68-61bf5dd4e7ea None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:10.926574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:10.926574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:10.926574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:10.926574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:10.926574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:10.926574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:10.926574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:10.926574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:10.926574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:10.926574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:10.926574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:10.926574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:10.926574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:10.926574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:10.926574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:10.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:10.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:10.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:10.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:10.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:10.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:10.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:10.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:10.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:10.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:10.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:10.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:10.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:10.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:10.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:10.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:10.928735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:10.928735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:10.928735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:10.928735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:10.928735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:10.928735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:10.928735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:10.928735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:10.928735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:10.928735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:10.928735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:10.928735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:10.930538 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 62/128] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:10.935676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bce3596-5670-43ca-aa68-61bf5dd4e7ea None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:11.009989 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:11.023161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86353d00-13e4-4a84-90c2-d1a1385132b8 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:11.063602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bce3596-5670-43ca-aa68-61bf5dd4e7ea None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:11.068949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f28fd7-03b8-455f-bf88-a5056b90678c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:11.069699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f28fd7-03b8-455f-bf88-a5056b90678c None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:11.070050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f28fd7-03b8-455f-bf88-a5056b90678c None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:11.070392 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f28fd7-03b8-455f-bf88-a5056b90678c None admin] PATH_INFO: `/v3/users/01782590dd294bd488a6d5c08a043ce1` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:11.071643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f28fd7-03b8-455f-bf88-a5056b90678c None admin] RBAC: Authorizing `identity:get_user(user_id=01782590dd294bd488a6d5c08a043ce1)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:11.073642 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f28fd7-03b8-455f-bf88-a5056b90678c None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:11.076654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 67/129] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:10 2022] GET /identity/v3/users/01782590dd294bd488a6d5c08a043ce1 => generated 247 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:11.077546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f28fd7-03b8-455f-bf88-a5056b90678c None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:11.106852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3849ee9b-7ee1-4802-b369-f3111c12b5b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:11.107852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3849ee9b-7ee1-4802-b369-f3111c12b5b8 None admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:11.108189 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3849ee9b-7ee1-4802-b369-f3111c12b5b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:11.108735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3849ee9b-7ee1-4802-b369-f3111c12b5b8 None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a/users/d4a141c6519544a4ad26f923f1fb0382/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:11.111869 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3849ee9b-7ee1-4802-b369-f3111c12b5b8 None admin] RBAC: Authorizing `identity:create_grant(project_id=1193eb16a76d432faf6141e47cc99c5a, user_id=d4a141c6519544a4ad26f923f1fb0382, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:11.114293 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3849ee9b-7ee1-4802-b369-f3111c12b5b8 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:11.139834 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:11.187458 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f28fd7-03b8-455f-bf88-a5056b90678c None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:11.191348 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 63/130] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:46:10 2022] PUT /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a/users/d4a141c6519544a4ad26f923f1fb0382/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:46:11.192719 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3849ee9b-7ee1-4802-b369-f3111c12b5b8 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:11.262964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9b2caa-1616-4eac-b645-131d40919c8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:11.263616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9b2caa-1616-4eac-b645-131d40919c8d None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:11.263906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9b2caa-1616-4eac-b645-131d40919c8d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:11.264189 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9b2caa-1616-4eac-b645-131d40919c8d None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a/users/d4a141c6519544a4ad26f923f1fb0382/roles/c599b217579e42a49c657c642ea0648f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:11.271261 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9b2caa-1616-4eac-b645-131d40919c8d None admin] RBAC: Authorizing `identity:create_grant(project_id=1193eb16a76d432faf6141e47cc99c5a, user_id=d4a141c6519544a4ad26f923f1fb0382, role_id=c599b217579e42a49c657c642ea0648f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:11.272791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9b2caa-1616-4eac-b645-131d40919c8d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:11.334745 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3849ee9b-7ee1-4802-b369-f3111c12b5b8 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:11.353896 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:11.378552 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 68/131] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:46:11 2022] PUT /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a/users/d4a141c6519544a4ad26f923f1fb0382/roles/c599b217579e42a49c657c642ea0648f => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:46:11.379504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d656c582-6227-4581-80f0-facabc74b476 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:11.381005 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d656c582-6227-4581-80f0-facabc74b476 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:11.382130 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9b2caa-1616-4eac-b645-131d40919c8d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:11.383015 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d656c582-6227-4581-80f0-facabc74b476 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:11.383901 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d656c582-6227-4581-80f0-facabc74b476 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:11.385718 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d656c582-6227-4581-80f0-facabc74b476 None admin] RBAC: Adding query filter params (name=member) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:11.387397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d656c582-6227-4581-80f0-facabc74b476 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:11.389036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d656c582-6227-4581-80f0-facabc74b476 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:11.408427 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 64/132] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:11 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:11.416531 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d656c582-6227-4581-80f0-facabc74b476 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:11.553223 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9b2caa-1616-4eac-b645-131d40919c8d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:11.569556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:11.603405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75353e37-9497-4e4e-ab36-19e980f27d9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:11.604080 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75353e37-9497-4e4e-ab36-19e980f27d9c None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:11.604354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75353e37-9497-4e4e-ab36-19e980f27d9c None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:11.604630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75353e37-9497-4e4e-ab36-19e980f27d9c None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:11.605858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75353e37-9497-4e4e-ab36-19e980f27d9c None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:11.608010 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75353e37-9497-4e4e-ab36-19e980f27d9c None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:11.610556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 69/133] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:11 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:11.612620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e69fe96-4db6-428a-86d0-15055f662640 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:11.612921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e69fe96-4db6-428a-86d0-15055f662640 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:11.613698 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e69fe96-4db6-428a-86d0-15055f662640 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:11.615464 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 70/134] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:11.617484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e69fe96-4db6-428a-86d0-15055f662640 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:11.642661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d656c582-6227-4581-80f0-facabc74b476 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:11.687345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53313ad-11a1-4d63-ab91-6d8453746095 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:11.687996 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53313ad-11a1-4d63-ab91-6d8453746095 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:11.688280 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53313ad-11a1-4d63-ab91-6d8453746095 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:11.688559 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53313ad-11a1-4d63-ab91-6d8453746095 None admin] PATH_INFO: `/v3/users/d4a141c6519544a4ad26f923f1fb0382` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:11.689734 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53313ad-11a1-4d63-ab91-6d8453746095 None admin] RBAC: Authorizing `identity:get_user(user_id=d4a141c6519544a4ad26f923f1fb0382)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:11.691201 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53313ad-11a1-4d63-ab91-6d8453746095 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:11.693554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 65/135] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:11 2022] GET /identity/v3/users/d4a141c6519544a4ad26f923f1fb0382 => generated 275 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:11.694696 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd3a8a1-fa97-4a22-943b-f7a2924c5c80 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:11.695061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd3a8a1-fa97-4a22-943b-f7a2924c5c80 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:11.695338 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd3a8a1-fa97-4a22-943b-f7a2924c5c80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:11.730103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-7dd3a8a1-fa97-4a22-943b-f7a2924c5c80 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:11.731318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7dd3a8a1-fa97-4a22-943b-f7a2924c5c80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:11.749818 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e69fe96-4db6-428a-86d0-15055f662640 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:11.812727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37935891-b77b-42df-98dc-6c9936e46091 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:11.813480 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37935891-b77b-42df-98dc-6c9936e46091 None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:11.813763 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37935891-b77b-42df-98dc-6c9936e46091 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:11.814052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37935891-b77b-42df-98dc-6c9936e46091 None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a/users/01782590dd294bd488a6d5c08a043ce1/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:11.818349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37935891-b77b-42df-98dc-6c9936e46091 None admin] RBAC: Authorizing `identity:create_grant(project_id=1193eb16a76d432faf6141e47cc99c5a, user_id=01782590dd294bd488a6d5c08a043ce1, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:11.822459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37935891-b77b-42df-98dc-6c9936e46091 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:11.824506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:11.853728 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 66/136] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:11.854425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd3a8a1-fa97-4a22-943b-f7a2924c5c80 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:11.919858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 71/137] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:46:11 2022] PUT /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a/users/01782590dd294bd488a6d5c08a043ce1/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 00:46:11.988201 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd3a8a1-fa97-4a22-943b-f7a2924c5c80 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:12.035564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc55666f-d2d3-4e9b-81bf-176e3e6ff8f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:12.036341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc55666f-d2d3-4e9b-81bf-176e3e6ff8f0 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:12.036615 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc55666f-d2d3-4e9b-81bf-176e3e6ff8f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:12.036890 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc55666f-d2d3-4e9b-81bf-176e3e6ff8f0 None admin] PATH_INFO: `/v3/projects/3cd6beb90b3c45c5bb9adeac3df910e1` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:12.038220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc55666f-d2d3-4e9b-81bf-176e3e6ff8f0 None admin] RBAC: Authorizing `identity:get_project(project_id=3cd6beb90b3c45c5bb9adeac3df910e1)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:12.039730 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc55666f-d2d3-4e9b-81bf-176e3e6ff8f0 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:12.041633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 67/138] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:11 2022] GET /identity/v3/projects/3cd6beb90b3c45c5bb9adeac3df910e1 => generated 312 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:12.047554 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37935891-b77b-42df-98dc-6c9936e46091 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:12.123568 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:12.195306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37935891-b77b-42df-98dc-6c9936e46091 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:12.234440 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39fce99-f4c7-4070-8da5-b0b9dea0f1e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:12.235400 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39fce99-f4c7-4070-8da5-b0b9dea0f1e0 None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:12.235860 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39fce99-f4c7-4070-8da5-b0b9dea0f1e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:12.236333 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39fce99-f4c7-4070-8da5-b0b9dea0f1e0 None admin] PATH_INFO: `/v3/projects/3cd6beb90b3c45c5bb9adeac3df910e1/users/d4a141c6519544a4ad26f923f1fb0382/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:12.242381 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39fce99-f4c7-4070-8da5-b0b9dea0f1e0 None admin] RBAC: Authorizing `identity:create_grant(project_id=3cd6beb90b3c45c5bb9adeac3df910e1, user_id=d4a141c6519544a4ad26f923f1fb0382, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:12.244030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39fce99-f4c7-4070-8da5-b0b9dea0f1e0 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:12.306919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc55666f-d2d3-4e9b-81bf-176e3e6ff8f0 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:12.321292 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 72/139] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:46:12 2022] PUT /identity/v3/projects/3cd6beb90b3c45c5bb9adeac3df910e1/users/d4a141c6519544a4ad26f923f1fb0382/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:46:12.363362 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:12.440329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc55666f-d2d3-4e9b-81bf-176e3e6ff8f0 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:12.493923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa69a60-e5e1-44ef-a00f-0e0b93d07153 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:12.494637 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa69a60-e5e1-44ef-a00f-0e0b93d07153 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:12.494891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa69a60-e5e1-44ef-a00f-0e0b93d07153 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:12.495250 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa69a60-e5e1-44ef-a00f-0e0b93d07153 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:12.502006 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa69a60-e5e1-44ef-a00f-0e0b93d07153 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:12.503522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa69a60-e5e1-44ef-a00f-0e0b93d07153 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:12.505505 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 68/140] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:46:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:12.511247 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39fce99-f4c7-4070-8da5-b0b9dea0f1e0 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:12.651083 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:12.662098 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39fce99-f4c7-4070-8da5-b0b9dea0f1e0 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:12.702764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a75822b-441e-4b00-b5a0-d4682c6d320f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:12.703282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a75822b-441e-4b00-b5a0-d4682c6d320f None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:12.703558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a75822b-441e-4b00-b5a0-d4682c6d320f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:12.703831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a75822b-441e-4b00-b5a0-d4682c6d320f None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:12.704496 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a75822b-441e-4b00-b5a0-d4682c6d320f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:12.705981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a75822b-441e-4b00-b5a0-d4682c6d320f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:12.828304 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 73/141] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:46:12 2022] POST /identity/v3/users => generated 283 bytes in 318 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:46:13.259383 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0778f5ce-697b-4fd2-9ef6-6118862e7ed9 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:13.259753 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0778f5ce-697b-4fd2-9ef6-6118862e7ed9 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:13.259942 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0778f5ce-697b-4fd2-9ef6-6118862e7ed9 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:13.261340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 69/142] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:13.266648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f8454-ed33-42bb-a17e-176c9f73f005 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:13.267088 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f8454-ed33-42bb-a17e-176c9f73f005 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:13.267292 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f8454-ed33-42bb-a17e-176c9f73f005 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:13.289497 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-3e6f8454-ed33-42bb-a17e-176c9f73f005 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:13.290716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-3e6f8454-ed33-42bb-a17e-176c9f73f005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:13.384182 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 74/143] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:13.800608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0778f5ce-697b-4fd2-9ef6-6118862e7ed9 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:13.881432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78440dfd-777f-4bc7-b402-8c5033ad34e0 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:13.882066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78440dfd-777f-4bc7-b402-8c5033ad34e0 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:13.882538 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78440dfd-777f-4bc7-b402-8c5033ad34e0 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:13.884105 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 75/144] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:46:13.890424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa04c11f-b641-42ed-b9ab-8b98a7590c43 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:13.890985 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa04c11f-b641-42ed-b9ab-8b98a7590c43 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:13.891459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa04c11f-b641-42ed-b9ab-8b98a7590c43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:13.933519 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0778f5ce-697b-4fd2-9ef6-6118862e7ed9 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:13.934270 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-aa04c11f-b641-42ed-b9ab-8b98a7590c43 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:13.936303 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-aa04c11f-b641-42ed-b9ab-8b98a7590c43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:13.953939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:13.995898 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1cc0da-5d40-47fc-8f1c-ba7e04299fff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:13.996552 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1cc0da-5d40-47fc-8f1c-ba7e04299fff None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:13.996828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1cc0da-5d40-47fc-8f1c-ba7e04299fff None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:13.997402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1cc0da-5d40-47fc-8f1c-ba7e04299fff None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:14.023052 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1cc0da-5d40-47fc-8f1c-ba7e04299fff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:14.030228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-4b1cc0da-5d40-47fc-8f1c-ba7e04299fff None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:14.030228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:14.030228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:14.030228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:14.030228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:14.030228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:14.030228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:14.030228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:14.030228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:14.030228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:14.030228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:14.030228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:14.030228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:14.030228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:14.030228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:14.030228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:14.031085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:14.031085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:14.031085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:14.031085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:14.031085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:14.031085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:14.031085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:14.031085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:14.031085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:14.031085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:14.031085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:14.031085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:14.031085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:14.031085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:14.031085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:14.031085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:14.031821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:14.031821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:14.031821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:14.031821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:14.031821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:14.031821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:14.031821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:14.031821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:14.031821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:14.031821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:14.031821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:14.031821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:14.033111 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 70/145] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:13 2022] GET /identity/v3/roles/reader => generated 84 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:14.037968 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64216f47-73ac-4456-969d-4a6feba3b33c None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:14.038265 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64216f47-73ac-4456-969d-4a6feba3b33c None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:14.038541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64216f47-73ac-4456-969d-4a6feba3b33c None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:14.039871 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 71/146] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:46:14.040578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64216f47-73ac-4456-969d-4a6feba3b33c None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:14.065954 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 76/147] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:14.068388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e32d3e0-ef5e-46d9-a1d8-84aebd607c9f None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:14.069147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e32d3e0-ef5e-46d9-a1d8-84aebd607c9f None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:14.069609 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e32d3e0-ef5e-46d9-a1d8-84aebd607c9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:14.092430 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-6e32d3e0-ef5e-46d9-a1d8-84aebd607c9f None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:14.093996 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6e32d3e0-ef5e-46d9-a1d8-84aebd607c9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:14.109427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:14.238568 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 77/148] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:14.250974 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64216f47-73ac-4456-969d-4a6feba3b33c None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:14.300348 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa1adb4-7d02-4d7a-b691-fb41033ba927 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:14.300971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa1adb4-7d02-4d7a-b691-fb41033ba927 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:14.301279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa1adb4-7d02-4d7a-b691-fb41033ba927 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:14.302186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa1adb4-7d02-4d7a-b691-fb41033ba927 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:14.302879 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa1adb4-7d02-4d7a-b691-fb41033ba927 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:14.303233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa1adb4-7d02-4d7a-b691-fb41033ba927 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:14.304722 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa1adb4-7d02-4d7a-b691-fb41033ba927 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:14.320369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 72/149] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:14 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:14.330226 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e32d3e0-ef5e-46d9-a1d8-84aebd607c9f None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:14.475099 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e32d3e0-ef5e-46d9-a1d8-84aebd607c9f None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:14.521321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8163bb8b-c12f-4bf2-b2f2-0737a8da0028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:14.521965 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8163bb8b-c12f-4bf2-b2f2-0737a8da0028 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:14.522248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8163bb8b-c12f-4bf2-b2f2-0737a8da0028 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:14.522522 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8163bb8b-c12f-4bf2-b2f2-0737a8da0028 None admin] PATH_INFO: `/v3/users/381b26af45ac4d8d967cfbe0d8639173` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:14.523748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8163bb8b-c12f-4bf2-b2f2-0737a8da0028 None admin] RBAC: Authorizing `identity:get_user(user_id=381b26af45ac4d8d967cfbe0d8639173)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:14.525217 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8163bb8b-c12f-4bf2-b2f2-0737a8da0028 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:14.526814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 78/150] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:14 2022] GET /identity/v3/users/381b26af45ac4d8d967cfbe0d8639173 => generated 289 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:14.532477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa1adb4-7d02-4d7a-b691-fb41033ba927 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:14.575260 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:14.620294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8163bb8b-c12f-4bf2-b2f2-0737a8da0028 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:14.647636 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:14.670850 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa1adb4-7d02-4d7a-b691-fb41033ba927 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:14.718729 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76dc5c6e-50bf-48e9-b7e5-0eda5fbc32c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:14.719828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76dc5c6e-50bf-48e9-b7e5-0eda5fbc32c9 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:14.720141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76dc5c6e-50bf-48e9-b7e5-0eda5fbc32c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:14.720511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76dc5c6e-50bf-48e9-b7e5-0eda5fbc32c9 None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:14.725822 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76dc5c6e-50bf-48e9-b7e5-0eda5fbc32c9 None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:14.727320 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76dc5c6e-50bf-48e9-b7e5-0eda5fbc32c9 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:14.729263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 73/151] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:14 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:14.729843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76dc5c6e-50bf-48e9-b7e5-0eda5fbc32c9 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:14.810647 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8163bb8b-c12f-4bf2-b2f2-0737a8da0028 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:14.882595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4311317f-d175-4b6a-9bce-ead3c59a336e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:14.883166 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4311317f-d175-4b6a-9bce-ead3c59a336e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:14.883447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4311317f-d175-4b6a-9bce-ead3c59a336e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:14.883722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4311317f-d175-4b6a-9bce-ead3c59a336e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:14.913733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4311317f-d175-4b6a-9bce-ead3c59a336e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:14.919244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-4311317f-d175-4b6a-9bce-ead3c59a336e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:14.919244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:14.919244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:14.919244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:14.919244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:14.919244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:14.919244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:14.919244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:14.919244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:14.919244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:14.919244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:14.919244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:14.919244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:14.919244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:14.919244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:14.919244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:14.920091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:14.920091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:14.920091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:14.920091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:14.920091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:14.920091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:14.920091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:14.920091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:14.920091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:14.920091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:14.920091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:14.920091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:14.920091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:14.920091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:14.920091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:14.920091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:14.920850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:14.920850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:14.920850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:14.920850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:14.920850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:14.920850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:14.920850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:14.920850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:14.920850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:14.920850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:14.920850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:14.920850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:14.920850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 79/152] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:14 2022] GET /identity/v3/roles/member => generated 84 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:46:14.921511 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4311317f-d175-4b6a-9bce-ead3c59a336e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:14.937704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76dc5c6e-50bf-48e9-b7e5-0eda5fbc32c9 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:14.939476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:14.992448 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6feaa52c-33e3-4d78-9bbf-632bbc6cb3f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:14.993101 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feaa52c-33e3-4d78-9bbf-632bbc6cb3f4 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:14.993464 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feaa52c-33e3-4d78-9bbf-632bbc6cb3f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:14.993735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6feaa52c-33e3-4d78-9bbf-632bbc6cb3f4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:15.010195 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6feaa52c-33e3-4d78-9bbf-632bbc6cb3f4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:15.015062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-6feaa52c-33e3-4d78-9bbf-632bbc6cb3f4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:15.015062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:15.015062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:15.015062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:15.015062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:15.015062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:15.015062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:15.015062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:15.015062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:15.015062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:15.015062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:15.015062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:15.015062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:15.015062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:15.015062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:15.015062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:15.016071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:15.016071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:15.016071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:15.016071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:15.016071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:15.016071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:15.016071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:15.016071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:15.016071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:15.016071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:15.016071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:15.016071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:15.016071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:15.016071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:15.016071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:15.016071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:15.016911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:15.016911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:15.016911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:15.016911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:15.016911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:15.016911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:15.016911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:15.016911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:15.016911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:15.016911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:15.016911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:15.016911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:15.016911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 74/153] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:46:14 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:15.017604 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6feaa52c-33e3-4d78-9bbf-632bbc6cb3f4 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:15.098434 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:15.162952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4311317f-d175-4b6a-9bce-ead3c59a336e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:15.193906 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6feaa52c-33e3-4d78-9bbf-632bbc6cb3f4 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:15.237354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e605c4-c82b-4c2d-8504-445baf0a85b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:15.238040 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e605c4-c82b-4c2d-8504-445baf0a85b2 None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:15.238323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e605c4-c82b-4c2d-8504-445baf0a85b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:15.238603 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e605c4-c82b-4c2d-8504-445baf0a85b2 None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a/users/381b26af45ac4d8d967cfbe0d8639173/roles/6b0968db45c64a66b001a39d40579011` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:15.246408 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e605c4-c82b-4c2d-8504-445baf0a85b2 None admin] RBAC: Authorizing `identity:create_grant(project_id=1193eb16a76d432faf6141e47cc99c5a, user_id=381b26af45ac4d8d967cfbe0d8639173, role_id=6b0968db45c64a66b001a39d40579011)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:15.248200 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e605c4-c82b-4c2d-8504-445baf0a85b2 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:15.250657 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35367d66-862b-406e-8d5e-25cea53adea4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:15.251288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35367d66-862b-406e-8d5e-25cea53adea4 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:15.251565 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35367d66-862b-406e-8d5e-25cea53adea4 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:15.251837 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35367d66-862b-406e-8d5e-25cea53adea4 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:15.252510 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35367d66-862b-406e-8d5e-25cea53adea4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:15.252807 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35367d66-862b-406e-8d5e-25cea53adea4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:15.254424 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35367d66-862b-406e-8d5e-25cea53adea4 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:15.269134 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 75/154] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:15.270000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35367d66-862b-406e-8d5e-25cea53adea4 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:15.298905 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:15.350411 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 80/155] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:46:14 2022] PUT /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a/users/381b26af45ac4d8d967cfbe0d8639173/roles/6b0968db45c64a66b001a39d40579011 => generated 0 bytes in 429 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:46:15.351474 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dfdb4d-17ee-445d-b0d3-356da9e55081 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:15.351762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dfdb4d-17ee-445d-b0d3-356da9e55081 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:15.352039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25dfdb4d-17ee-445d-b0d3-356da9e55081 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:15.357800 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 81/156] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:15 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:15.358472 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dfdb4d-17ee-445d-b0d3-356da9e55081 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:15.391902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35367d66-862b-406e-8d5e-25cea53adea4 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:15.450914 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3b08d9-46b9-4ec4-ae70-ceea32c62a39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:15.451460 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3b08d9-46b9-4ec4-ae70-ceea32c62a39 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:15.451738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3b08d9-46b9-4ec4-ae70-ceea32c62a39 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:15.452010 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3b08d9-46b9-4ec4-ae70-ceea32c62a39 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:15.452683 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3b08d9-46b9-4ec4-ae70-ceea32c62a39 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:15.452984 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3b08d9-46b9-4ec4-ae70-ceea32c62a39 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:15.454411 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3b08d9-46b9-4ec4-ae70-ceea32c62a39 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:15.475037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 76/157] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:46:15 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:15.476743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72204a4d-ce57-492e-b06f-1b4b9ccd213c None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:15.476743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72204a4d-ce57-492e-b06f-1b4b9ccd213c None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:15.476943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72204a4d-ce57-492e-b06f-1b4b9ccd213c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:15.504552 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-72204a4d-ce57-492e-b06f-1b4b9ccd213c None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:15.506338 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-72204a4d-ce57-492e-b06f-1b4b9ccd213c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:15.635679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25dfdb4d-17ee-445d-b0d3-356da9e55081 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:15.646230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 77/158] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:15.647933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72204a4d-ce57-492e-b06f-1b4b9ccd213c None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:15.675949 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:15.678604 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f04f7d-d6ae-46ad-9141-45299688d473 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:15.679334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f04f7d-d6ae-46ad-9141-45299688d473 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:15.679692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f04f7d-d6ae-46ad-9141-45299688d473 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:15.680039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f04f7d-d6ae-46ad-9141-45299688d473 None admin] PATH_INFO: `/v3/users/d4a141c6519544a4ad26f923f1fb0382` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:15.681324 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f04f7d-d6ae-46ad-9141-45299688d473 None admin] RBAC: Authorizing `identity:get_user(user_id=d4a141c6519544a4ad26f923f1fb0382)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:15.682940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f04f7d-d6ae-46ad-9141-45299688d473 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:15.684626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 82/159] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:15 2022] GET /identity/v3/users/d4a141c6519544a4ad26f923f1fb0382 => generated 275 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:15.695623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f04f7d-d6ae-46ad-9141-45299688d473 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:15.698760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:15.811471 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72204a4d-ce57-492e-b06f-1b4b9ccd213c None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:15.866160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85f6d47-f4f2-4d79-9d27-913b6d670716 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:15.866811 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85f6d47-f4f2-4d79-9d27-913b6d670716 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:15.867088 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85f6d47-f4f2-4d79-9d27-913b6d670716 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:15.867361 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85f6d47-f4f2-4d79-9d27-913b6d670716 None admin] PATH_INFO: `/v3/users/d4a141c6519544a4ad26f923f1fb0382` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:15.871074 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85f6d47-f4f2-4d79-9d27-913b6d670716 None admin] RBAC: Authorizing `identity:get_user(user_id=d4a141c6519544a4ad26f923f1fb0382)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:15.872809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85f6d47-f4f2-4d79-9d27-913b6d670716 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:15.875155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 78/160] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:15 2022] GET /identity/v3/users/d4a141c6519544a4ad26f923f1fb0382 => generated 275 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:15.881672 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85f6d47-f4f2-4d79-9d27-913b6d670716 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:15.896700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f04f7d-d6ae-46ad-9141-45299688d473 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:16.007392 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e817e67-be6d-404b-ba9e-f48d5571db80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:16.008428 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e817e67-be6d-404b-ba9e-f48d5571db80 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:16.008925 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e817e67-be6d-404b-ba9e-f48d5571db80 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:16.009445 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e817e67-be6d-404b-ba9e-f48d5571db80 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:16.063080 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:16.107356 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85f6d47-f4f2-4d79-9d27-913b6d670716 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:16.182687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89d9820-9cae-401d-b0d2-f73040d929f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:16.183465 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89d9820-9cae-401d-b0d2-f73040d929f5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:16.183858 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89d9820-9cae-401d-b0d2-f73040d929f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:16.184133 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89d9820-9cae-401d-b0d2-f73040d929f5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:16.194232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e817e67-be6d-404b-ba9e-f48d5571db80 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:16.195719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e817e67-be6d-404b-ba9e-f48d5571db80 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:16.197705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:16.329936 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89d9820-9cae-401d-b0d2-f73040d929f5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:16.331336 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89d9820-9cae-401d-b0d2-f73040d929f5 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:16.377563 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 83/161] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 682 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 04 00:46:16.380778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201b30f4-e8ef-440e-b8eb-2d6456fbc234 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:16.381321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201b30f4-e8ef-440e-b8eb-2d6456fbc234 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:16.381665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201b30f4-e8ef-440e-b8eb-2d6456fbc234 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:16.383167 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 84/162] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:16 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:16.386225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201b30f4-e8ef-440e-b8eb-2d6456fbc234 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:16.542935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:16.588736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 79/163] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 707 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 04 00:46:16.589455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89d9820-9cae-401d-b0d2-f73040d929f5 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:16.622869 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201b30f4-e8ef-440e-b8eb-2d6456fbc234 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:16.624949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:16.705477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cfc031-98d3-452c-b98b-93f217a36838 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:16.706987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cfc031-98d3-452c-b98b-93f217a36838 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:16.708223 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cfc031-98d3-452c-b98b-93f217a36838 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:16.708796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cfc031-98d3-452c-b98b-93f217a36838 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:16.736207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3cfc031-98d3-452c-b98b-93f217a36838 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:16.744801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-f3cfc031-98d3-452c-b98b-93f217a36838 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:16.744801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:16.744801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:16.744801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:16.744801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:16.744801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:16.744801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:16.744801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:16.744801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:16.744801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:16.744801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:16.744801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:16.744801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:16.744801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:16.744801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:16.744801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:16.745882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:16.745882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:16.745882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:16.745882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:16.745882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:16.745882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:16.745882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:16.745882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:16.745882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:16.745882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:16.745882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:16.745882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:16.745882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:16.745882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:16.745882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:16.745882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:16.746688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:16.746688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:16.746688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:16.746688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:16.746688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:16.746688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:16.746688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:16.746688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:16.746688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:16.746688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:16.746688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:16.746688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:16.753740 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 85/164] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:46:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 370 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:16.755342 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a43690-b8ef-4b55-b830-0bb0d975e14a None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:16.755911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a43690-b8ef-4b55-b830-0bb0d975e14a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:16.756398 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a43690-b8ef-4b55-b830-0bb0d975e14a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:16.786661 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-d1a43690-b8ef-4b55-b830-0bb0d975e14a None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:16.787172 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89d9820-9cae-401d-b0d2-f73040d929f5 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:16.788651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d1a43690-b8ef-4b55-b830-0bb0d975e14a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:16.860942 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0b7a42-d775-405f-8e99-37cbd5e8fa22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:16.862019 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0b7a42-d775-405f-8e99-37cbd5e8fa22 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:16.862503 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0b7a42-d775-405f-8e99-37cbd5e8fa22 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:16.862977 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0b7a42-d775-405f-8e99-37cbd5e8fa22 None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:16.866467 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0b7a42-d775-405f-8e99-37cbd5e8fa22 None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:16.868187 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0b7a42-d775-405f-8e99-37cbd5e8fa22 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:16.871711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 80/165] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:16 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:16.872966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0b7a42-d775-405f-8e99-37cbd5e8fa22 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:16.917012 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 86/166] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:16.918220 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a43690-b8ef-4b55-b830-0bb0d975e14a None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:16.976590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:17.066363 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0b7a42-d775-405f-8e99-37cbd5e8fa22 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:17.107115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:17.132263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c28a1a-9ad1-4222-92b8-46457421a505 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:17.133283 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c28a1a-9ad1-4222-92b8-46457421a505 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:17.133792 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c28a1a-9ad1-4222-92b8-46457421a505 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:17.134277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c28a1a-9ad1-4222-92b8-46457421a505 None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:17.139332 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a43690-b8ef-4b55-b830-0bb0d975e14a None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:17.140320 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c28a1a-9ad1-4222-92b8-46457421a505 None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:17.142165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c28a1a-9ad1-4222-92b8-46457421a505 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:17.144088 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 81/167] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:16 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:17.145101 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c28a1a-9ad1-4222-92b8-46457421a505 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:17.183532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f409f2c6-b701-4c8e-b41d-ae829aa95849 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:17.184289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f409f2c6-b701-4c8e-b41d-ae829aa95849 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:17.184576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f409f2c6-b701-4c8e-b41d-ae829aa95849 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:17.184855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f409f2c6-b701-4c8e-b41d-ae829aa95849 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:17.185570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f409f2c6-b701-4c8e-b41d-ae829aa95849 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:17.185878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f409f2c6-b701-4c8e-b41d-ae829aa95849 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:17.187417 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f409f2c6-b701-4c8e-b41d-ae829aa95849 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:17.197702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 87/168] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:17.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f409f2c6-b701-4c8e-b41d-ae829aa95849 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:17.325260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f409f2c6-b701-4c8e-b41d-ae829aa95849 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:17.360129 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c28a1a-9ad1-4222-92b8-46457421a505 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:17.367202 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06dfd0b3-0366-4fb3-9e8e-ca23e50887d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:17.367827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dfd0b3-0366-4fb3-9e8e-ca23e50887d2 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:17.368108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dfd0b3-0366-4fb3-9e8e-ca23e50887d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:17.368383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06dfd0b3-0366-4fb3-9e8e-ca23e50887d2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:17.369163 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06dfd0b3-0366-4fb3-9e8e-ca23e50887d2 None admin] RBAC: Adding query filter params (role.id=c599b217579e42a49c657c642ea0648f, scope.project.id=1193eb16a76d432faf6141e47cc99c5a, user.id=d4a141c6519544a4ad26f923f1fb0382) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:17.369481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06dfd0b3-0366-4fb3-9e8e-ca23e50887d2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:17.370923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06dfd0b3-0366-4fb3-9e8e-ca23e50887d2 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:17.380646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 88/169] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:17 2022] GET /identity/v3/role_assignments?user.id=d4a141c6519544a4ad26f923f1fb0382&scope.project.id=1193eb16a76d432faf6141e47cc99c5a&role.id=c599b217579e42a49c657c642ea0648f => generated 608 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:17.381737 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06dfd0b3-0366-4fb3-9e8e-ca23e50887d2 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:17.408315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:17.419964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:17.448934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d36a313-7d5d-4406-83b4-3c69e83fa7f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:17.450012 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d36a313-7d5d-4406-83b4-3c69e83fa7f8 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:17.450561 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d36a313-7d5d-4406-83b4-3c69e83fa7f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:17.451037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d36a313-7d5d-4406-83b4-3c69e83fa7f8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:17.452011 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d36a313-7d5d-4406-83b4-3c69e83fa7f8 None admin] RBAC: Adding query filter params (role.id=4bfd6d0544ee4a9e933538e78135a12c, scope.project.id=1193eb16a76d432faf6141e47cc99c5a, user.id=d4a141c6519544a4ad26f923f1fb0382) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:17.452540 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d36a313-7d5d-4406-83b4-3c69e83fa7f8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:17.454242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d36a313-7d5d-4406-83b4-3c69e83fa7f8 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:17.467083 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 82/170] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:17 2022] GET /identity/v3/role_assignments?user.id=d4a141c6519544a4ad26f923f1fb0382&scope.project.id=1193eb16a76d432faf6141e47cc99c5a&role.id=4bfd6d0544ee4a9e933538e78135a12c => generated 608 bytes in 322 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:17.468078 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d36a313-7d5d-4406-83b4-3c69e83fa7f8 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:17.637842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06dfd0b3-0366-4fb3-9e8e-ca23e50887d2 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:17.687432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b276ada2-7dab-4e52-aaf2-12f7714f0750 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:17.688044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b276ada2-7dab-4e52-aaf2-12f7714f0750 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:17.688326 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b276ada2-7dab-4e52-aaf2-12f7714f0750 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:17.688605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b276ada2-7dab-4e52-aaf2-12f7714f0750 None admin] PATH_INFO: `/v3/users/01782590dd294bd488a6d5c08a043ce1` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:17.690094 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b276ada2-7dab-4e52-aaf2-12f7714f0750 None admin] RBAC: Authorizing `identity:get_user(user_id=01782590dd294bd488a6d5c08a043ce1)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:17.691815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b276ada2-7dab-4e52-aaf2-12f7714f0750 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:17.694956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 89/171] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:17 2022] GET /identity/v3/users/01782590dd294bd488a6d5c08a043ce1 => generated 247 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:17.696742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e125a3-da86-4330-abbf-7ab13dcda92a None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:17.697036 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e125a3-da86-4330-abbf-7ab13dcda92a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:17.697340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e125a3-da86-4330-abbf-7ab13dcda92a None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:17.699003 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 90/172] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:17.704091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e125a3-da86-4330-abbf-7ab13dcda92a None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:17.705766 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d36a313-7d5d-4406-83b4-3c69e83fa7f8 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:17.790213 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c12292a-d695-4d54-8df5-2e260233f2db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:17.790877 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c12292a-d695-4d54-8df5-2e260233f2db None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:17.791143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c12292a-d695-4d54-8df5-2e260233f2db None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:17.791418 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c12292a-d695-4d54-8df5-2e260233f2db None admin] PATH_INFO: `/v3/roles/member` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:17.815226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c12292a-d695-4d54-8df5-2e260233f2db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:17.823455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-9c12292a-d695-4d54-8df5-2e260233f2db None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:17.823455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:17.823455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:17.823455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:17.823455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:17.823455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:17.823455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:17.823455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:17.823455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:17.823455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:17.823455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:17.823455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:17.823455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:17.823455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:17.823455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:17.823455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:17.824346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:17.824346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:17.824346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:17.824346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:17.824346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:17.824346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:17.824346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:17.824346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:17.824346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:17.824346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:17.824346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:17.824346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:17.824346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:17.824346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:17.824346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:17.824346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:17.825158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:17.825158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:17.825158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:17.825158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:17.825158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:17.825158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:17.825158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:17.825158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:17.825158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:17.825158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:17.825158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:17.825158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:17.829800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 83/173] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 362 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:17.831091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfbb78d-2302-470d-9648-26e72dccba69 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:17.831373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfbb78d-2302-470d-9648-26e72dccba69 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:17.831647 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfbb78d-2302-470d-9648-26e72dccba69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:17.852808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-fbfbb78d-2302-470d-9648-26e72dccba69 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:17.856217 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-fbfbb78d-2302-470d-9648-26e72dccba69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:17.874627 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:17.899406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e125a3-da86-4330-abbf-7ab13dcda92a None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:17.908929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:17.944563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 84/174] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:17.946034 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4126d170-a757-49fb-afcf-b2b8e6c2764e None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:17.946420 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4126d170-a757-49fb-afcf-b2b8e6c2764e None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:17.946695 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4126d170-a757-49fb-afcf-b2b8e6c2764e None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:17.946864 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0260b4b5-74dd-4b9f-8891-ca4736af65e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:17.947459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0260b4b5-74dd-4b9f-8891-ca4736af65e3 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:17.947742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0260b4b5-74dd-4b9f-8891-ca4736af65e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:17.947933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 85/175] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:17.948283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0260b4b5-74dd-4b9f-8891-ca4736af65e3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:17.948620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4126d170-a757-49fb-afcf-b2b8e6c2764e None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:18.043839 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0260b4b5-74dd-4b9f-8891-ca4736af65e3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:18.046219 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0260b4b5-74dd-4b9f-8891-ca4736af65e3 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:18.103013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4126d170-a757-49fb-afcf-b2b8e6c2764e None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:18.144729 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c46dca-c26b-4903-8bc3-83115a941e6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:18.145790 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c46dca-c26b-4903-8bc3-83115a941e6c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:18.146328 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c46dca-c26b-4903-8bc3-83115a941e6c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:18.146888 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c46dca-c26b-4903-8bc3-83115a941e6c None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:18.147769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c46dca-c26b-4903-8bc3-83115a941e6c None admin] RBAC: Adding query filter params (name=member) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:18.148258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c46dca-c26b-4903-8bc3-83115a941e6c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:18.149846 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c46dca-c26b-4903-8bc3-83115a941e6c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:18.159090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 86/176] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:17 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:18.165110 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504edb13-3c0e-4c1b-a3a5-30666e06a459 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:18.165659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504edb13-3c0e-4c1b-a3a5-30666e06a459 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:18.166142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504edb13-3c0e-4c1b-a3a5-30666e06a459 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:18.166994 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:18.169196 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 87/177] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:18 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:18.170539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ab8d86-9a3c-4da3-aa58-a76286faba10 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:18.171078 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ab8d86-9a3c-4da3-aa58-a76286faba10 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:18.171559 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ab8d86-9a3c-4da3-aa58-a76286faba10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:18.186680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 91/178] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 483 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:18.189289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0260b4b5-74dd-4b9f-8891-ca4736af65e3 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:18.196285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-c3ab8d86-9a3c-4da3-aa58-a76286faba10 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:18.197791 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c3ab8d86-9a3c-4da3-aa58-a76286faba10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:18.230747 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:18.285666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 88/179] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:18.291862 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbdd605-6669-4545-8162-b17f276c523f None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:18.292151 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbdd605-6669-4545-8162-b17f276c523f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:18.292431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbdd605-6669-4545-8162-b17f276c523f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:18.310250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0260b4b5-74dd-4b9f-8891-ca4736af65e3 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:18.318102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-5dbdd605-6669-4545-8162-b17f276c523f None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:18.319368 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-5dbdd605-6669-4545-8162-b17f276c523f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:18.348831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2fc6a6-6c84-4040-8af8-4eeb19fe1579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:18.350106 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2fc6a6-6c84-4040-8af8-4eeb19fe1579 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:18.350602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2fc6a6-6c84-4040-8af8-4eeb19fe1579 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:18.351074 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2fc6a6-6c84-4040-8af8-4eeb19fe1579 None admin] PATH_INFO: `/v3/users/d4a141c6519544a4ad26f923f1fb0382` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:18.352514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2fc6a6-6c84-4040-8af8-4eeb19fe1579 None admin] RBAC: Authorizing `identity:get_user(user_id=d4a141c6519544a4ad26f923f1fb0382)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:18.354229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2fc6a6-6c84-4040-8af8-4eeb19fe1579 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:18.356081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 92/180] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:18 2022] GET /identity/v3/users/d4a141c6519544a4ad26f923f1fb0382 => generated 275 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:18.361633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2fc6a6-6c84-4040-8af8-4eeb19fe1579 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:18.395535 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 89/181] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:18.396215 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbdd605-6669-4545-8162-b17f276c523f None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:18.423144 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:18.471450 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2fc6a6-6c84-4040-8af8-4eeb19fe1579 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:18.502031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:18.509514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbdd605-6669-4545-8162-b17f276c523f None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:18.541401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f7879e-cabd-4586-a2d8-03b98926fcab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:18.542040 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f7879e-cabd-4586-a2d8-03b98926fcab None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:18.542314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f7879e-cabd-4586-a2d8-03b98926fcab None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:18.542581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f7879e-cabd-4586-a2d8-03b98926fcab None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:18.545351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f7879e-cabd-4586-a2d8-03b98926fcab None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:18.546766 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f7879e-cabd-4586-a2d8-03b98926fcab None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:18.548923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 93/182] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:18 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:18.549302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7302694-85ad-4705-9098-4e7a0c7f9f14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:18.549727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f7879e-cabd-4586-a2d8-03b98926fcab None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:18.550482 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7302694-85ad-4705-9098-4e7a0c7f9f14 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:18.550955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7302694-85ad-4705-9098-4e7a0c7f9f14 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:18.551439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7302694-85ad-4705-9098-4e7a0c7f9f14 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:18.563716 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7302694-85ad-4705-9098-4e7a0c7f9f14 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:18.570026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-e7302694-85ad-4705-9098-4e7a0c7f9f14 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:18.570026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:18.570026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:18.570026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:18.570026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:18.570026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:18.570026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:18.570026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:18.570026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:18.570026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:18.570026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:18.570026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:18.570026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:18.570026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:18.570026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:18.570026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:18.570891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:18.570891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:18.570891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:18.570891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:18.570891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:18.570891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:18.570891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:18.570891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:18.570891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:18.570891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:18.570891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:18.570891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:18.570891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:18.570891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:18.570891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:18.570891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:18.571663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:18.571663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:18.571663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:18.571663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:18.571663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:18.571663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:18.571663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:18.571663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:18.571663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:18.571663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:18.571663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:18.571663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:18.574165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 90/183] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:46:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:18.575130 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7302694-85ad-4705-9098-4e7a0c7f9f14 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:18.679529 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f7879e-cabd-4586-a2d8-03b98926fcab None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:18.694077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7302694-85ad-4705-9098-4e7a0c7f9f14 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:18.704883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:18.717122 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e77dac7-cea6-4950-9fd5-11da2d351976 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:18.717765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e77dac7-cea6-4950-9fd5-11da2d351976 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:18.718116 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e77dac7-cea6-4950-9fd5-11da2d351976 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:18.718517 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e77dac7-cea6-4950-9fd5-11da2d351976 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:18.730443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a078c4ad-57b9-4f62-90ad-274333d68d0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:18.731157 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a078c4ad-57b9-4f62-90ad-274333d68d0d None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:18.731445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a078c4ad-57b9-4f62-90ad-274333d68d0d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:18.731722 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a078c4ad-57b9-4f62-90ad-274333d68d0d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:18.732462 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a078c4ad-57b9-4f62-90ad-274333d68d0d None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=1193eb16a76d432faf6141e47cc99c5a, user.id=01782590dd294bd488a6d5c08a043ce1) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:18.732773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a078c4ad-57b9-4f62-90ad-274333d68d0d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:18.734286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a078c4ad-57b9-4f62-90ad-274333d68d0d None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:18.740873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 91/184] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:18 2022] GET /identity/v3/role_assignments?user.id=01782590dd294bd488a6d5c08a043ce1&scope.project.id=1193eb16a76d432faf6141e47cc99c5a&role.id=19c006412bec45dc8febd7cc86b9678f => generated 608 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:18.742224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c06bb4e-cc6c-4763-bd65-379bb54a7817 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:18.742516 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c06bb4e-cc6c-4763-bd65-379bb54a7817 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:18.742799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c06bb4e-cc6c-4763-bd65-379bb54a7817 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:18.744155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 92/185] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:18.744783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c06bb4e-cc6c-4763-bd65-379bb54a7817 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:18.775850 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:18.798979 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e77dac7-cea6-4950-9fd5-11da2d351976 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:18.800474 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e77dac7-cea6-4950-9fd5-11da2d351976 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:18.866426 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c06bb4e-cc6c-4763-bd65-379bb54a7817 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:18.907460 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e53f2a-a883-4306-a138-71fa031901de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:18.908068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e53f2a-a883-4306-a138-71fa031901de None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:18.908340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e53f2a-a883-4306-a138-71fa031901de None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:18.908609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e53f2a-a883-4306-a138-71fa031901de None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:18.909308 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e53f2a-a883-4306-a138-71fa031901de None admin] RBAC: Adding query filter params (name=admin) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:18.909609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e53f2a-a883-4306-a138-71fa031901de None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:18.911193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e53f2a-a883-4306-a138-71fa031901de None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:18.919746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 94/186] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:18.919925 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 93/187] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:18.920906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e77dac7-cea6-4950-9fd5-11da2d351976 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:18.921712 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e53f2a-a883-4306-a138-71fa031901de None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:18.950259 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:19.014210 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:19.031673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e77dac7-cea6-4950-9fd5-11da2d351976 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:19.031933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e53f2a-a883-4306-a138-71fa031901de None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:19.065699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02dc577f-770c-4305-af7b-9d693cb71278 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:19.065900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c3e980-93bd-42a8-ba66-f8d88752f03c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:19.066335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dc577f-770c-4305-af7b-9d693cb71278 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:19.066462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c3e980-93bd-42a8-ba66-f8d88752f03c None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:19.066619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dc577f-770c-4305-af7b-9d693cb71278 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:19.066746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c3e980-93bd-42a8-ba66-f8d88752f03c None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:19.066898 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dc577f-770c-4305-af7b-9d693cb71278 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:19.067019 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c3e980-93bd-42a8-ba66-f8d88752f03c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:19.068222 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c3e980-93bd-42a8-ba66-f8d88752f03c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:19.069714 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c3e980-93bd-42a8-ba66-f8d88752f03c None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:19.071274 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 95/188] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:46:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:19.072372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49ec81f-5aae-4793-abdd-a0ee9fe36a22 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:19.072675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49ec81f-5aae-4793-abdd-a0ee9fe36a22 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:19.072922 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49ec81f-5aae-4793-abdd-a0ee9fe36a22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:19.077658 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02dc577f-770c-4305-af7b-9d693cb71278 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:19.082497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-02dc577f-770c-4305-af7b-9d693cb71278 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:19.082497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:19.082497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:19.082497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:19.082497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:19.082497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:19.082497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:19.082497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:19.082497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:19.082497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:19.082497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:19.082497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:19.082497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:19.082497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:19.082497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:19.082497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:19.084430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:19.084430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:19.084430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:19.084430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:19.084430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:19.084430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:19.084430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:19.084430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:19.084430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:19.084430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:19.084430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:19.084430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:19.084430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:19.084430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:19.084430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:19.084430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:19.085259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:19.085259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:19.085259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:19.085259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:19.085259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:19.085259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:19.085259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:19.085259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:19.085259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:19.085259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:19.085259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:19.085259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:19.085259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 94/189] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:46:18 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:19.085259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02dc577f-770c-4305-af7b-9d693cb71278 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:19.090871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-a49ec81f-5aae-4793-abdd-a0ee9fe36a22 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:19.092055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a49ec81f-5aae-4793-abdd-a0ee9fe36a22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:19.158345 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 96/190] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:19.159161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49ec81f-5aae-4793-abdd-a0ee9fe36a22 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:19.182014 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:19.193858 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02dc577f-770c-4305-af7b-9d693cb71278 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:19.229812 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b42667a-259a-4bbc-864d-7ee02e3ad127 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:19.230445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b42667a-259a-4bbc-864d-7ee02e3ad127 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:19.230728 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b42667a-259a-4bbc-864d-7ee02e3ad127 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:19.231002 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b42667a-259a-4bbc-864d-7ee02e3ad127 None admin] PATH_INFO: `/v3/projects/3cd6beb90b3c45c5bb9adeac3df910e1` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:19.232136 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b42667a-259a-4bbc-864d-7ee02e3ad127 None admin] RBAC: Authorizing `identity:get_project(project_id=3cd6beb90b3c45c5bb9adeac3df910e1)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:19.233620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b42667a-259a-4bbc-864d-7ee02e3ad127 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:19.240587 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 95/191] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:19 2022] GET /identity/v3/projects/3cd6beb90b3c45c5bb9adeac3df910e1 => generated 312 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:19.241365 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b42667a-259a-4bbc-864d-7ee02e3ad127 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:19.254792 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:19.269743 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49ec81f-5aae-4793-abdd-a0ee9fe36a22 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:19.307044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f386b93b-4e1b-4927-9fc4-c95b00849034 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:19.307684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f386b93b-4e1b-4927-9fc4-c95b00849034 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:19.307966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f386b93b-4e1b-4927-9fc4-c95b00849034 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:19.308243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f386b93b-4e1b-4927-9fc4-c95b00849034 None admin] PATH_INFO: `/v3/users/1d534fbaeec64a2bbaf523be4f6cf886` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:19.321465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f386b93b-4e1b-4927-9fc4-c95b00849034 None admin] RBAC: Authorizing `identity:get_user(user_id=1d534fbaeec64a2bbaf523be4f6cf886)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:19.322909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f386b93b-4e1b-4927-9fc4-c95b00849034 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:19.325113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 97/192] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:19 2022] GET /identity/v3/users/1d534fbaeec64a2bbaf523be4f6cf886 => generated 283 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:19.325621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f386b93b-4e1b-4927-9fc4-c95b00849034 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:19.352864 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b42667a-259a-4bbc-864d-7ee02e3ad127 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:19.388448 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dbea8d-98f6-4340-8fa8-fff2c79c776b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:19.389179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dbea8d-98f6-4340-8fa8-fff2c79c776b None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:19.389535 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dbea8d-98f6-4340-8fa8-fff2c79c776b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:19.389918 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dbea8d-98f6-4340-8fa8-fff2c79c776b None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:19.390631 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dbea8d-98f6-4340-8fa8-fff2c79c776b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:19.392109 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dbea8d-98f6-4340-8fa8-fff2c79c776b None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:19.417030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:19.438540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f386b93b-4e1b-4927-9fc4-c95b00849034 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:19.473029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ec0f3c-2f0e-4f54-b027-e6f39a4b0c2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:19.473684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ec0f3c-2f0e-4f54-b027-e6f39a4b0c2d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:19.473972 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ec0f3c-2f0e-4f54-b027-e6f39a4b0c2d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:19.474243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ec0f3c-2f0e-4f54-b027-e6f39a4b0c2d None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:19.474909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ec0f3c-2f0e-4f54-b027-e6f39a4b0c2d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:19.475214 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ec0f3c-2f0e-4f54-b027-e6f39a4b0c2d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:19.476682 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ec0f3c-2f0e-4f54-b027-e6f39a4b0c2d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:19.484369 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 98/193] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:46:19 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:19.485186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ec0f3c-2f0e-4f54-b027-e6f39a4b0c2d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:19.499364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 96/194] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:46:19 2022] POST /identity/v3/users => generated 297 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:46:19.499364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dbea8d-98f6-4340-8fa8-fff2c79c776b None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:19.579816 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:19.603828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ec0f3c-2f0e-4f54-b027-e6f39a4b0c2d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:19.618360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dbea8d-98f6-4340-8fa8-fff2c79c776b None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:19.642923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b18d6d-fa37-4795-bb20-239f8a1ccdf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:19.643550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b18d6d-fa37-4795-bb20-239f8a1ccdf2 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:19.643839 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b18d6d-fa37-4795-bb20-239f8a1ccdf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:19.644088 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b18d6d-fa37-4795-bb20-239f8a1ccdf2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:19.644854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b18d6d-fa37-4795-bb20-239f8a1ccdf2 None admin] RBAC: Adding query filter params (role.id=4bfd6d0544ee4a9e933538e78135a12c, scope.project.id=3cd6beb90b3c45c5bb9adeac3df910e1, user.id=d4a141c6519544a4ad26f923f1fb0382) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:19.645757 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b18d6d-fa37-4795-bb20-239f8a1ccdf2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:19.647353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b18d6d-fa37-4795-bb20-239f8a1ccdf2 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:19.653649 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 99/195] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:19 2022] GET /identity/v3/role_assignments?user.id=d4a141c6519544a4ad26f923f1fb0382&scope.project.id=3cd6beb90b3c45c5bb9adeac3df910e1&role.id=4bfd6d0544ee4a9e933538e78135a12c => generated 608 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:19.661966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b18d6d-fa37-4795-bb20-239f8a1ccdf2 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:19.661966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:19.662145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab793e59-8168-467c-9322-7626238ca9c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:19.662768 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab793e59-8168-467c-9322-7626238ca9c7 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:19.663135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab793e59-8168-467c-9322-7626238ca9c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:19.663439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab793e59-8168-467c-9322-7626238ca9c7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:19.766701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab793e59-8168-467c-9322-7626238ca9c7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:19.768234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab793e59-8168-467c-9322-7626238ca9c7 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:19.810992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b18d6d-fa37-4795-bb20-239f8a1ccdf2 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:19.862271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53938182-9319-4ba3-a29d-5b40e3a5a85a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:19.862738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:19.863693 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53938182-9319-4ba3-a29d-5b40e3a5a85a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:19.864144 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53938182-9319-4ba3-a29d-5b40e3a5a85a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:19.864690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53938182-9319-4ba3-a29d-5b40e3a5a85a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:19.882093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53938182-9319-4ba3-a29d-5b40e3a5a85a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:19.887626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-53938182-9319-4ba3-a29d-5b40e3a5a85a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:19.887626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:19.887626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:19.887626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:19.887626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:19.887626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:19.887626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:19.887626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:19.887626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:19.887626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:19.887626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:19.887626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:19.887626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:19.887626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:19.887626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:19.887626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:19.888567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:19.888567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:19.888567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:19.888567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:19.888567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:19.888567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:19.888567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:19.888567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:19.888567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:19.888567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:19.888567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:19.888567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:19.888567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:19.888567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:19.888567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:19.888567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:19.889748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:19.889748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:19.889748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:19.889748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:19.889748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:19.889748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:19.889748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:19.889748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:19.889748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:19.889748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:19.889748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:19.889748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:19.889748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 100/196] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:19 2022] GET /identity/v3/roles/reader => generated 84 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:19.889748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53938182-9319-4ba3-a29d-5b40e3a5a85a None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:19.915078 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 97/197] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:19.916392 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab793e59-8168-467c-9322-7626238ca9c7 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:19.981544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:20.008936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53938182-9319-4ba3-a29d-5b40e3a5a85a None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:20.056006 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71887fb9-5da5-4190-813c-fbee7b6d7a12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:20.056671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71887fb9-5da5-4190-813c-fbee7b6d7a12 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:20.056949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71887fb9-5da5-4190-813c-fbee7b6d7a12 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:20.057272 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71887fb9-5da5-4190-813c-fbee7b6d7a12 None admin] PATH_INFO: `/v3/users/1d534fbaeec64a2bbaf523be4f6cf886` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:20.058695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71887fb9-5da5-4190-813c-fbee7b6d7a12 None admin] RBAC: Authorizing `identity:get_user(user_id=1d534fbaeec64a2bbaf523be4f6cf886)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:20.060349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71887fb9-5da5-4190-813c-fbee7b6d7a12 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:20.062646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 101/198] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:19 2022] GET /identity/v3/users/1d534fbaeec64a2bbaf523be4f6cf886 => generated 283 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:20.067798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab793e59-8168-467c-9322-7626238ca9c7 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:20.067952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71887fb9-5da5-4190-813c-fbee7b6d7a12 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:20.113103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d6c7c3-f6a6-4e9c-a1a2-a3a956d56fa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:20.114544 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d6c7c3-f6a6-4e9c-a1a2-a3a956d56fa1 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:20.115471 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d6c7c3-f6a6-4e9c-a1a2-a3a956d56fa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:20.115939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d6c7c3-f6a6-4e9c-a1a2-a3a956d56fa1 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:20.117209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d6c7c3-f6a6-4e9c-a1a2-a3a956d56fa1 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:20.117970 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d6c7c3-f6a6-4e9c-a1a2-a3a956d56fa1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:20.119817 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d6c7c3-f6a6-4e9c-a1a2-a3a956d56fa1 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:20.131531 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 98/199] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:19 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:20.132978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d6c7c3-f6a6-4e9c-a1a2-a3a956d56fa1 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:20.179356 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71887fb9-5da5-4190-813c-fbee7b6d7a12 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:20.206384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:20.218542 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:20.225317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53428b64-5fbf-45b3-8a7f-24848fd06fad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:20.227098 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53428b64-5fbf-45b3-8a7f-24848fd06fad None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:20.231699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53428b64-5fbf-45b3-8a7f-24848fd06fad None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:20.231699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53428b64-5fbf-45b3-8a7f-24848fd06fad None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:20.231699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53428b64-5fbf-45b3-8a7f-24848fd06fad None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:20.231699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53428b64-5fbf-45b3-8a7f-24848fd06fad None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:20.233269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 102/200] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:20 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:20.234181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53428b64-5fbf-45b3-8a7f-24848fd06fad None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:20.264596 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d6c7c3-f6a6-4e9c-a1a2-a3a956d56fa1 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:20.299543 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7bc660-1176-49f6-8958-14c77dc5752e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:20.300304 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7bc660-1176-49f6-8958-14c77dc5752e None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:20.300883 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7bc660-1176-49f6-8958-14c77dc5752e None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:20.301179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7bc660-1176-49f6-8958-14c77dc5752e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:20.364417 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53428b64-5fbf-45b3-8a7f-24848fd06fad None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:20.378439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7bc660-1176-49f6-8958-14c77dc5752e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:20.380098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7bc660-1176-49f6-8958-14c77dc5752e None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:20.408338 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:20.454587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ded2e3-803e-4ddd-9b66-ce50c04026ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:20.455234 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ded2e3-803e-4ddd-9b66-ce50c04026ba None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:20.455652 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ded2e3-803e-4ddd-9b66-ce50c04026ba None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:20.455994 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ded2e3-803e-4ddd-9b66-ce50c04026ba None admin] PATH_INFO: `/v3/users/381b26af45ac4d8d967cfbe0d8639173` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:20.457327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ded2e3-803e-4ddd-9b66-ce50c04026ba None admin] RBAC: Authorizing `identity:get_user(user_id=381b26af45ac4d8d967cfbe0d8639173)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:20.462341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ded2e3-803e-4ddd-9b66-ce50c04026ba None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:20.464559 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 103/201] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:20 2022] GET /identity/v3/users/381b26af45ac4d8d967cfbe0d8639173 => generated 289 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:20.465847 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ded2e3-803e-4ddd-9b66-ce50c04026ba None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:20.503998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 99/202] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:20.505376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7bc660-1176-49f6-8958-14c77dc5752e None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:20.542180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:20.579042 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ded2e3-803e-4ddd-9b66-ce50c04026ba None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:20.616045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696c9519-ff9d-4b1e-afb9-dc512a42aaea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:20.616660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696c9519-ff9d-4b1e-afb9-dc512a42aaea None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:20.616982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696c9519-ff9d-4b1e-afb9-dc512a42aaea None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:20.617390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696c9519-ff9d-4b1e-afb9-dc512a42aaea None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:20.618192 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696c9519-ff9d-4b1e-afb9-dc512a42aaea None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=d737f411ca514fb4aa93ee11d55926ae, user.id=1d534fbaeec64a2bbaf523be4f6cf886) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:20.618566 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696c9519-ff9d-4b1e-afb9-dc512a42aaea None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:20.620304 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696c9519-ff9d-4b1e-afb9-dc512a42aaea None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:20.630243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7bc660-1176-49f6-8958-14c77dc5752e None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:20.631911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 104/203] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:20 2022] GET /identity/v3/role_assignments?user.id=1d534fbaeec64a2bbaf523be4f6cf886&scope.project.id=d737f411ca514fb4aa93ee11d55926ae&role.id=19c006412bec45dc8febd7cc86b9678f => generated 259 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:20.633149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696c9519-ff9d-4b1e-afb9-dc512a42aaea None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:20.669106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:20.686831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc4e0ff-6fe2-45b9-ae5b-37c2d7766e39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:20.687455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc4e0ff-6fe2-45b9-ae5b-37c2d7766e39 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:20.687727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc4e0ff-6fe2-45b9-ae5b-37c2d7766e39 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:20.687995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc4e0ff-6fe2-45b9-ae5b-37c2d7766e39 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:20.812599 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc4e0ff-6fe2-45b9-ae5b-37c2d7766e39 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:20.814214 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc4e0ff-6fe2-45b9-ae5b-37c2d7766e39 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:20.841567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696c9519-ff9d-4b1e-afb9-dc512a42aaea None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:20.909765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:20.923429 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013fd67b-da7a-4305-9f4c-f82e27d10edc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:20.924162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013fd67b-da7a-4305-9f4c-f82e27d10edc None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:20.924447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013fd67b-da7a-4305-9f4c-f82e27d10edc None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:20.924719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013fd67b-da7a-4305-9f4c-f82e27d10edc None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:20.925954 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013fd67b-da7a-4305-9f4c-f82e27d10edc None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:20.928334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013fd67b-da7a-4305-9f4c-f82e27d10edc None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:20.935031 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 105/204] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:20 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:20.936243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013fd67b-da7a-4305-9f4c-f82e27d10edc None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:20.937117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 100/205] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:20.942302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc4e0ff-6fe2-45b9-ae5b-37c2d7766e39 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:20.981436 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:21.072424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013fd67b-da7a-4305-9f4c-f82e27d10edc None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:21.109350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42a56ad-e625-4057-b19c-6058aa3b9734 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:21.109989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42a56ad-e625-4057-b19c-6058aa3b9734 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:21.110828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42a56ad-e625-4057-b19c-6058aa3b9734 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:21.111104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42a56ad-e625-4057-b19c-6058aa3b9734 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:21.111849 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42a56ad-e625-4057-b19c-6058aa3b9734 None admin] RBAC: Adding query filter params (role.id=c599b217579e42a49c657c642ea0648f, scope.project.id=d737f411ca514fb4aa93ee11d55926ae, user.id=1d534fbaeec64a2bbaf523be4f6cf886) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:21.112158 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42a56ad-e625-4057-b19c-6058aa3b9734 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:21.113613 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a42a56ad-e625-4057-b19c-6058aa3b9734 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:21.120186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 106/206] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:20 2022] GET /identity/v3/role_assignments?user.id=1d534fbaeec64a2bbaf523be4f6cf886&scope.project.id=d737f411ca514fb4aa93ee11d55926ae&role.id=c599b217579e42a49c657c642ea0648f => generated 259 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:21.157648 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc4e0ff-6fe2-45b9-ae5b-37c2d7766e39 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:21.196704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66854726-afe0-4ecd-8719-cd13616aee2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:21.197294 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66854726-afe0-4ecd-8719-cd13616aee2d None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:21.197569 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66854726-afe0-4ecd-8719-cd13616aee2d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:21.197839 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66854726-afe0-4ecd-8719-cd13616aee2d None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:21.198998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66854726-afe0-4ecd-8719-cd13616aee2d None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:21.200454 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66854726-afe0-4ecd-8719-cd13616aee2d None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:21.202520 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 101/207] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:20 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:21.210281 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42a56ad-e625-4057-b19c-6058aa3b9734 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:21.295512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:21.352891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42a56ad-e625-4057-b19c-6058aa3b9734 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:21.398349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5003d82-9367-4070-a55f-958bddf3e159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:21.398957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5003d82-9367-4070-a55f-958bddf3e159 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:21.399239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5003d82-9367-4070-a55f-958bddf3e159 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:21.399501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5003d82-9367-4070-a55f-958bddf3e159 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:21.400247 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5003d82-9367-4070-a55f-958bddf3e159 None admin] RBAC: Adding query filter params (role.id=6b0968db45c64a66b001a39d40579011, scope.project.id=1193eb16a76d432faf6141e47cc99c5a, user.id=381b26af45ac4d8d967cfbe0d8639173) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:21.400552 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5003d82-9367-4070-a55f-958bddf3e159 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:21.402070 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5003d82-9367-4070-a55f-958bddf3e159 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:21.414381 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 107/208] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:21 2022] GET /identity/v3/role_assignments?user.id=381b26af45ac4d8d967cfbe0d8639173&scope.project.id=1193eb16a76d432faf6141e47cc99c5a&role.id=6b0968db45c64a66b001a39d40579011 => generated 608 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:21.835414 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb36b0f-ceef-4221-aa7f-5fcf882ae4fb None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:21.836117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb36b0f-ceef-4221-aa7f-5fcf882ae4fb None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:21.836117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb36b0f-ceef-4221-aa7f-5fcf882ae4fb None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:21.837629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 102/209] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:21.843185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dff4e8d-b5f8-4aa4-a487-89b25ed74982 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:21.843651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dff4e8d-b5f8-4aa4-a487-89b25ed74982 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:21.843728 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dff4e8d-b5f8-4aa4-a487-89b25ed74982 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:21.864653 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-3dff4e8d-b5f8-4aa4-a487-89b25ed74982 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:21.866239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-3dff4e8d-b5f8-4aa4-a487-89b25ed74982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:21.871600 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecfc623-b158-46f2-b49c-ba250d3db6f6 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:21.871878 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecfc623-b158-46f2-b49c-ba250d3db6f6 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:21.872415 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecfc623-b158-46f2-b49c-ba250d3db6f6 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:21.874061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 103/210] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:21.879162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4e8af9-fb5c-47ae-8d53-90a3a5719f79 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:21.879511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4e8af9-fb5c-47ae-8d53-90a3a5719f79 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:21.880467 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4e8af9-fb5c-47ae-8d53-90a3a5719f79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:21.903898 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9f4e8af9-fb5c-47ae-8d53-90a3a5719f79 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:21.905069 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9f4e8af9-fb5c-47ae-8d53-90a3a5719f79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:21.963799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 108/211] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:46:22.003553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 104/212] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:46:22.241920 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dff4e8d-b5f8-4aa4-a487-89b25ed74982 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:22.288276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4e8af9-fb5c-47ae-8d53-90a3a5719f79 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:22.311668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:22.320358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:22.357509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dff4e8d-b5f8-4aa4-a487-89b25ed74982 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:22.394919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac44f542-3af0-4332-acdc-c80907f0b2c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:22.396047 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac44f542-3af0-4332-acdc-c80907f0b2c9 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:22.396540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac44f542-3af0-4332-acdc-c80907f0b2c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:22.397023 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac44f542-3af0-4332-acdc-c80907f0b2c9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:22.410259 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac44f542-3af0-4332-acdc-c80907f0b2c9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:22.415608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4e8af9-fb5c-47ae-8d53-90a3a5719f79 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:22.416871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-ac44f542-3af0-4332-acdc-c80907f0b2c9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:22.416871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:22.416871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:22.416871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:22.416871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:22.416871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:22.416871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:22.416871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:22.416871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:22.416871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:22.416871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:22.416871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:22.416871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:22.416871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:22.416871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:22.416871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:22.417921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:22.417921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:22.417921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:22.417921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:22.417921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:22.417921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:22.417921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:22.417921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:22.417921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:22.417921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:22.417921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:22.417921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:22.417921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:22.417921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:22.417921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:22.417921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:22.418727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:22.418727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:22.418727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:22.418727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:22.418727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:22.418727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:22.418727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:22.418727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:22.418727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:22.418727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:22.418727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:22.418727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:22.425923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 109/213] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:22.427224 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac44f542-3af0-4332-acdc-c80907f0b2c9 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:22.468262 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148ad4bf-1e58-4e94-87d0-6dfcbf284135 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:22.468906 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148ad4bf-1e58-4e94-87d0-6dfcbf284135 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:22.469609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148ad4bf-1e58-4e94-87d0-6dfcbf284135 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:22.469895 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148ad4bf-1e58-4e94-87d0-6dfcbf284135 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:22.471276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148ad4bf-1e58-4e94-87d0-6dfcbf284135 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:22.472781 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148ad4bf-1e58-4e94-87d0-6dfcbf284135 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:22.480416 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 105/214] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:46:22 2022] GET /identity/v3/domains/default => generated 199 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:22.481415 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148ad4bf-1e58-4e94-87d0-6dfcbf284135 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:22.561315 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac44f542-3af0-4332-acdc-c80907f0b2c9 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:22.584235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:22.602441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de52b9f3-6324-49e6-910e-628594aedb37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:22.603361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de52b9f3-6324-49e6-910e-628594aedb37 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:22.603874 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de52b9f3-6324-49e6-910e-628594aedb37 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:22.604326 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de52b9f3-6324-49e6-910e-628594aedb37 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:22.605228 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de52b9f3-6324-49e6-910e-628594aedb37 None admin] RBAC: Adding query filter params (name=member) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:22.605732 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de52b9f3-6324-49e6-910e-628594aedb37 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:22.607295 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de52b9f3-6324-49e6-910e-628594aedb37 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:22.607720 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148ad4bf-1e58-4e94-87d0-6dfcbf284135 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:22.609432 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:22.617784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 110/215] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:22.624200 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de52b9f3-6324-49e6-910e-628594aedb37 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:22.647946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d3db78-c44f-466f-846c-6e2944bf1330 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:22.648622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d3db78-c44f-466f-846c-6e2944bf1330 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:22.648902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d3db78-c44f-466f-846c-6e2944bf1330 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:22.649319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d3db78-c44f-466f-846c-6e2944bf1330 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:22.650299 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d3db78-c44f-466f-846c-6e2944bf1330 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:22.657420 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d3db78-c44f-466f-846c-6e2944bf1330 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:22.735353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de52b9f3-6324-49e6-910e-628594aedb37 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:22.763670 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 106/216] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:46:22 2022] POST /identity/v3/users => generated 297 bytes in 282 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:46:22.784414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493211a5-5f62-487b-8723-d5f66c036155 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:22.785462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493211a5-5f62-487b-8723-d5f66c036155 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:22.786014 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493211a5-5f62-487b-8723-d5f66c036155 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:22.786481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493211a5-5f62-487b-8723-d5f66c036155 None admin] PATH_INFO: `/v3/users/9d0cf27ac9334ddfa7382dc6967ada80` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:22.804229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493211a5-5f62-487b-8723-d5f66c036155 None admin] RBAC: Authorizing `identity:get_user(user_id=9d0cf27ac9334ddfa7382dc6967ada80)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:22.806120 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493211a5-5f62-487b-8723-d5f66c036155 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:22.808095 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 111/217] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:22 2022] GET /identity/v3/users/9d0cf27ac9334ddfa7382dc6967ada80 => generated 297 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:22.814381 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d3db78-c44f-466f-846c-6e2944bf1330 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:22.945097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d3db78-c44f-466f-846c-6e2944bf1330 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:22.980772 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14caf9f2-b144-4df9-b31f-9d7774753d5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:22.981421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14caf9f2-b144-4df9-b31f-9d7774753d5f None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:22.981703 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14caf9f2-b144-4df9-b31f-9d7774753d5f None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:22.981978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14caf9f2-b144-4df9-b31f-9d7774753d5f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:22.984710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986cd9f0-15c9-4023-b82c-a7808efc803e None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:22.985718 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986cd9f0-15c9-4023-b82c-a7808efc803e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:22.986055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986cd9f0-15c9-4023-b82c-a7808efc803e None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:22.988555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 112/218] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:46:22.996328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9fabb9-e91b-45de-9d61-0b74deb47dd8 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:22.996687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9fabb9-e91b-45de-9d61-0b74deb47dd8 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:22.997723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9fabb9-e91b-45de-9d61-0b74deb47dd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:23.007864 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:23.021478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-9d9fabb9-e91b-45de-9d61-0b74deb47dd8 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:23.022917 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-9d9fabb9-e91b-45de-9d61-0b74deb47dd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:23.064074 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:23.076096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14caf9f2-b144-4df9-b31f-9d7774753d5f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:23.077654 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14caf9f2-b144-4df9-b31f-9d7774753d5f None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:23.114715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 113/219] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:23.195287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 107/220] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:23.202718 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9fabb9-e91b-45de-9d61-0b74deb47dd8 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:23.282831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a8e7e-b3e3-401c-972f-473ce7bbd1f7 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:23.283601 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a8e7e-b3e3-401c-972f-473ce7bbd1f7 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:23.284077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a8e7e-b3e3-401c-972f-473ce7bbd1f7 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:23.285695 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 108/221] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:23.290838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71640f51-e21d-4ea1-924c-bfeea20683b1 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:23.291396 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71640f51-e21d-4ea1-924c-bfeea20683b1 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:23.291865 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71640f51-e21d-4ea1-924c-bfeea20683b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:23.313305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-71640f51-e21d-4ea1-924c-bfeea20683b1 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:23.314769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-71640f51-e21d-4ea1-924c-bfeea20683b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:23.316363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9fabb9-e91b-45de-9d61-0b74deb47dd8 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:23.359899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e59d19d-fb98-4fc2-a6d9-0f3311fdd7dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:23.360923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e59d19d-fb98-4fc2-a6d9-0f3311fdd7dd None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:23.361593 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e59d19d-fb98-4fc2-a6d9-0f3311fdd7dd None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:23.362017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e59d19d-fb98-4fc2-a6d9-0f3311fdd7dd None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:23.363541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e59d19d-fb98-4fc2-a6d9-0f3311fdd7dd None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:23.366327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e59d19d-fb98-4fc2-a6d9-0f3311fdd7dd None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:23.368685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 114/222] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:23 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:23.375219 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e59d19d-fb98-4fc2-a6d9-0f3311fdd7dd None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:23.404923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:23.428475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:23.436546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 109/223] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:46:23.436546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71640f51-e21d-4ea1-924c-bfeea20683b1 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:23.522379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e59d19d-fb98-4fc2-a6d9-0f3311fdd7dd None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:23.584878 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71640f51-e21d-4ea1-924c-bfeea20683b1 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:23.606362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930cf5e3-bb40-43a2-be85-c661608dff05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:23.607001 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930cf5e3-bb40-43a2-be85-c661608dff05 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:23.607279 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930cf5e3-bb40-43a2-be85-c661608dff05 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:23.607545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-930cf5e3-bb40-43a2-be85-c661608dff05 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:23.608288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930cf5e3-bb40-43a2-be85-c661608dff05 None admin] RBAC: Adding query filter params (role.id=4bfd6d0544ee4a9e933538e78135a12c, scope.project.id=d737f411ca514fb4aa93ee11d55926ae, user.id=9d0cf27ac9334ddfa7382dc6967ada80) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:23.608595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930cf5e3-bb40-43a2-be85-c661608dff05 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:23.610120 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-930cf5e3-bb40-43a2-be85-c661608dff05 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:23.624443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f675a7-4d15-4443-bbdc-3215f5c10689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:23.625216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 115/224] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:23 2022] GET /identity/v3/role_assignments?user.id=9d0cf27ac9334ddfa7382dc6967ada80&scope.project.id=d737f411ca514fb4aa93ee11d55926ae&role.id=4bfd6d0544ee4a9e933538e78135a12c => generated 259 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:23.625662 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f675a7-4d15-4443-bbdc-3215f5c10689 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:23.626226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f675a7-4d15-4443-bbdc-3215f5c10689 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:23.626690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f675a7-4d15-4443-bbdc-3215f5c10689 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:23.639613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f675a7-4d15-4443-bbdc-3215f5c10689 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:23.645765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-f7f675a7-4d15-4443-bbdc-3215f5c10689 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:23.645765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:23.645765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:23.645765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:23.645765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:23.645765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:23.645765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:23.645765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:23.645765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:23.645765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:23.645765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:23.645765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:23.645765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:23.645765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:23.645765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:23.645765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:23.646779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:23.646779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:23.646779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:23.646779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:23.646779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:23.646779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:23.646779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:23.646779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:23.646779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:23.646779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:23.646779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:23.646779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:23.646779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:23.646779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:23.646779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:23.646779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:23.647714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:23.647714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:23.647714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:23.647714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:23.647714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:23.647714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:23.647714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:23.647714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:23.647714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:23.647714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:23.647714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:23.647714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:23.650704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 110/225] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:46:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:23.658239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930cf5e3-bb40-43a2-be85-c661608dff05 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:23.752542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f675a7-4d15-4443-bbdc-3215f5c10689 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:23.807402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:23.826177 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-930cf5e3-bb40-43a2-be85-c661608dff05 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:23.833570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:23.896689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6f3aef-193f-4d03-beef-639bb7936b0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:23.897268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6f3aef-193f-4d03-beef-639bb7936b0a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:23.897554 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6f3aef-193f-4d03-beef-639bb7936b0a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:23.897836 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6f3aef-193f-4d03-beef-639bb7936b0a None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:23.898521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6f3aef-193f-4d03-beef-639bb7936b0a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:23.898831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6f3aef-193f-4d03-beef-639bb7936b0a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:23.900201 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6f3aef-193f-4d03-beef-639bb7936b0a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:23.920540 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f675a7-4d15-4443-bbdc-3215f5c10689 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:23.922629 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 116/226] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:23.928566 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6f3aef-193f-4d03-beef-639bb7936b0a None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:24.017016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8940ac72-c00e-423b-b1b9-692b3ca0a30d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:24.017772 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8940ac72-c00e-423b-b1b9-692b3ca0a30d None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:24.017979 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8940ac72-c00e-423b-b1b9-692b3ca0a30d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:24.018320 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8940ac72-c00e-423b-b1b9-692b3ca0a30d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:24.046705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8940ac72-c00e-423b-b1b9-692b3ca0a30d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:24.056788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-8940ac72-c00e-423b-b1b9-692b3ca0a30d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:24.056788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:24.056788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:24.056788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:24.056788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:24.056788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:24.056788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:24.056788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:24.056788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:24.056788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:24.056788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:24.056788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:24.056788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:24.056788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:24.056788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:24.056788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:24.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:24.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:24.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:24.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:24.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:24.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:24.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:24.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:24.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:24.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:24.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:24.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:24.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:24.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:24.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:24.057685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:24.058461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:24.058461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:24.058461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:24.058461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:24.058461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:24.058461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:24.058461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:24.058461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:24.058461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:24.058461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:24.058461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:24.058461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:24.064075 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 111/227] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:46:23 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:46:24.064898 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8940ac72-c00e-423b-b1b9-692b3ca0a30d None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:24.075847 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6f3aef-193f-4d03-beef-639bb7936b0a None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:24.180167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:24.186115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8940ac72-c00e-423b-b1b9-692b3ca0a30d None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:24.201764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33c35a8-bfef-4a48-a2e1-173cc1a7b035 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:24.202421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33c35a8-bfef-4a48-a2e1-173cc1a7b035 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:24.202703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33c35a8-bfef-4a48-a2e1-173cc1a7b035 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:24.202977 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33c35a8-bfef-4a48-a2e1-173cc1a7b035 None admin] PATH_INFO: `/v3/users/1d534fbaeec64a2bbaf523be4f6cf886` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:24.204161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33c35a8-bfef-4a48-a2e1-173cc1a7b035 None admin] RBAC: Authorizing `identity:get_user(user_id=1d534fbaeec64a2bbaf523be4f6cf886)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:24.207213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33c35a8-bfef-4a48-a2e1-173cc1a7b035 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:24.208844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 117/228] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:23 2022] GET /identity/v3/users/1d534fbaeec64a2bbaf523be4f6cf886 => generated 283 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:24.214530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33c35a8-bfef-4a48-a2e1-173cc1a7b035 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:24.257462 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb65d2b4-6d17-4f10-bb70-9359e35dd8e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:24.258082 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb65d2b4-6d17-4f10-bb70-9359e35dd8e0 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:24.258362 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb65d2b4-6d17-4f10-bb70-9359e35dd8e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:24.258634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb65d2b4-6d17-4f10-bb70-9359e35dd8e0 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:24.259311 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb65d2b4-6d17-4f10-bb70-9359e35dd8e0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:24.259629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb65d2b4-6d17-4f10-bb70-9359e35dd8e0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:24.264621 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb65d2b4-6d17-4f10-bb70-9359e35dd8e0 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:24.276942 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:24.280334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 112/229] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:46:24 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:24.286030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb65d2b4-6d17-4f10-bb70-9359e35dd8e0 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:24.385394 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33c35a8-bfef-4a48-a2e1-173cc1a7b035 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:24.401522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb65d2b4-6d17-4f10-bb70-9359e35dd8e0 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:24.441713 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf68bd6-656f-4c3e-9a61-07c1c93f9217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:24.442719 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf68bd6-656f-4c3e-9a61-07c1c93f9217 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:24.443209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf68bd6-656f-4c3e-9a61-07c1c93f9217 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:24.443816 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf68bd6-656f-4c3e-9a61-07c1c93f9217 None admin] PATH_INFO: `/v3/users/1d534fbaeec64a2bbaf523be4f6cf886` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:24.449838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf68bd6-656f-4c3e-9a61-07c1c93f9217 None admin] RBAC: Authorizing `identity:get_user(user_id=1d534fbaeec64a2bbaf523be4f6cf886)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:24.451507 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf68bd6-656f-4c3e-9a61-07c1c93f9217 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:24.454866 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 113/230] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:24 2022] GET /identity/v3/users/1d534fbaeec64a2bbaf523be4f6cf886 => generated 283 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:24.461588 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf68bd6-656f-4c3e-9a61-07c1c93f9217 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:24.500915 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812cdd84-941a-4082-af6c-fda62cbc7bca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:24.501636 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812cdd84-941a-4082-af6c-fda62cbc7bca None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:24.501915 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812cdd84-941a-4082-af6c-fda62cbc7bca None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:24.502186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812cdd84-941a-4082-af6c-fda62cbc7bca None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:24.503459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812cdd84-941a-4082-af6c-fda62cbc7bca None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:24.505650 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812cdd84-941a-4082-af6c-fda62cbc7bca None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:24.507453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 118/231] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:24 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:24.514707 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812cdd84-941a-4082-af6c-fda62cbc7bca None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:24.534882 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:24.630033 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:24.634080 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812cdd84-941a-4082-af6c-fda62cbc7bca None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:24.673999 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0300bd-74f6-4c21-bb3c-bc471926a601 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:24.674669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0300bd-74f6-4c21-bb3c-bc471926a601 None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:24.674945 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0300bd-74f6-4c21-bb3c-bc471926a601 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:24.675213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0300bd-74f6-4c21-bb3c-bc471926a601 None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae/users/1d534fbaeec64a2bbaf523be4f6cf886/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:24.684662 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0300bd-74f6-4c21-bb3c-bc471926a601 None admin] RBAC: Authorizing `identity:create_grant(project_id=d737f411ca514fb4aa93ee11d55926ae, user_id=1d534fbaeec64a2bbaf523be4f6cf886, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:24.686286 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0300bd-74f6-4c21-bb3c-bc471926a601 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:24.703073 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf68bd6-656f-4c3e-9a61-07c1c93f9217 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:24.738070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b2dbee-b9f8-4e29-99cd-d24edff35b6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:24.738696 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b2dbee-b9f8-4e29-99cd-d24edff35b6f None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:24.739577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b2dbee-b9f8-4e29-99cd-d24edff35b6f None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:24.739804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b2dbee-b9f8-4e29-99cd-d24edff35b6f None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:24.740904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b2dbee-b9f8-4e29-99cd-d24edff35b6f None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:24.742416 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b2dbee-b9f8-4e29-99cd-d24edff35b6f None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:24.744052 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 114/232] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:24 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:24.750113 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b2dbee-b9f8-4e29-99cd-d24edff35b6f None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:24.770065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 119/233] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:46:24 2022] PUT /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae/users/1d534fbaeec64a2bbaf523be4f6cf886/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 00:46:24.881908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:24.893491 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b2dbee-b9f8-4e29-99cd-d24edff35b6f None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:24.945571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64fc9b7-e3f8-4e33-a6c8-44de1f8c8742 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:24.946264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64fc9b7-e3f8-4e33-a6c8-44de1f8c8742 None admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:24.946541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64fc9b7-e3f8-4e33-a6c8-44de1f8c8742 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:24.946862 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64fc9b7-e3f8-4e33-a6c8-44de1f8c8742 None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae/users/1d534fbaeec64a2bbaf523be4f6cf886/roles/c599b217579e42a49c657c642ea0648f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:24.949053 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64fc9b7-e3f8-4e33-a6c8-44de1f8c8742 None admin] RBAC: Authorizing `identity:create_grant(project_id=d737f411ca514fb4aa93ee11d55926ae, user_id=1d534fbaeec64a2bbaf523be4f6cf886, role_id=c599b217579e42a49c657c642ea0648f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:24.950517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64fc9b7-e3f8-4e33-a6c8-44de1f8c8742 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:25.067964 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 115/234] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:46:24 2022] PUT /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae/users/1d534fbaeec64a2bbaf523be4f6cf886/roles/c599b217579e42a49c657c642ea0648f => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 00:46:25.293445 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af40dc67-9808-4659-ab5b-298224c747fc None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:25.293977 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af40dc67-9808-4659-ab5b-298224c747fc None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:25.293977 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af40dc67-9808-4659-ab5b-298224c747fc None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:25.295407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 120/235] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:25.300987 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec98d7ba-27a2-42c8-8084-cef5c3244af9 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:25.301407 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec98d7ba-27a2-42c8-8084-cef5c3244af9 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:25.301760 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec98d7ba-27a2-42c8-8084-cef5c3244af9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:25.322699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-ec98d7ba-27a2-42c8-8084-cef5c3244af9 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:25.323809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ec98d7ba-27a2-42c8-8084-cef5c3244af9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:25.398420 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 116/236] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:25.467291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc36bb5-0425-4f8b-b304-b285606e6844 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:25.467580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc36bb5-0425-4f8b-b304-b285606e6844 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:25.467854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc36bb5-0425-4f8b-b304-b285606e6844 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:25.469505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 121/237] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:25.474448 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4232ce37-5d67-4aa3-9cbc-873430060b2b None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:25.474732 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4232ce37-5d67-4aa3-9cbc-873430060b2b None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:25.475007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4232ce37-5d67-4aa3-9cbc-873430060b2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:25.503392 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-4232ce37-5d67-4aa3-9cbc-873430060b2b None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:25.505224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4232ce37-5d67-4aa3-9cbc-873430060b2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:25.593179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 117/238] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:25.773702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc36bb5-0425-4f8b-b304-b285606e6844 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:25.905826 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc36bb5-0425-4f8b-b304-b285606e6844 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:25.925120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88b7eae-2617-4598-800e-543198dda518 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:25.925120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88b7eae-2617-4598-800e-543198dda518 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:25.925120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88b7eae-2617-4598-800e-543198dda518 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:25.925988 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 118/239] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:46:25.933418 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543a8f40-96dd-4ee7-9f69-d4103475d5d9 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:25.933951 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543a8f40-96dd-4ee7-9f69-d4103475d5d9 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:25.934531 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543a8f40-96dd-4ee7-9f69-d4103475d5d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:25.954708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ff582a-4de2-4545-b2f7-8d3a95a05c29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:25.955361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ff582a-4de2-4545-b2f7-8d3a95a05c29 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:25.955638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ff582a-4de2-4545-b2f7-8d3a95a05c29 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:25.955907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ff582a-4de2-4545-b2f7-8d3a95a05c29 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:25.966105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-543a8f40-96dd-4ee7-9f69-d4103475d5d9 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:25.967294 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-543a8f40-96dd-4ee7-9f69-d4103475d5d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:25.968336 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ff582a-4de2-4545-b2f7-8d3a95a05c29 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:25.974814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-41ff582a-4de2-4545-b2f7-8d3a95a05c29 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:25.974814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:25.974814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:25.974814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:25.974814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:25.974814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:25.974814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:25.974814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:25.974814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:25.974814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:25.974814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:25.974814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:25.974814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:25.974814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:25.974814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:25.974814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:25.975685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:25.975685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:25.975685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:25.975685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:25.975685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:25.975685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:25.975685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:25.975685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:25.975685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:25.975685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:25.975685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:25.975685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:25.975685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:25.975685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:25.975685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:25.975685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:25.976471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:25.976471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:25.976471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:25.976471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:25.976471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:25.976471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:25.976471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:25.976471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:25.976471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:25.976471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:25.976471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:25.976471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:25.977058 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 122/240] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:25 2022] GET /identity/v3/roles/reader => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:25.977618 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ff582a-4de2-4545-b2f7-8d3a95a05c29 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:25.988285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:26.003322 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:26.046591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 119/241] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:46:26.047450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543a8f40-96dd-4ee7-9f69-d4103475d5d9 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:26.086149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ff582a-4de2-4545-b2f7-8d3a95a05c29 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:26.122039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cef6122-7ec7-4246-86f1-45875122e8ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:26.123241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cef6122-7ec7-4246-86f1-45875122e8ec None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:26.123728 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cef6122-7ec7-4246-86f1-45875122e8ec None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:26.124138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cef6122-7ec7-4246-86f1-45875122e8ec None admin] PATH_INFO: `/v3/domains/default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:26.125419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cef6122-7ec7-4246-86f1-45875122e8ec None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:26.127006 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cef6122-7ec7-4246-86f1-45875122e8ec None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:26.128723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 123/242] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:46:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:26.139220 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cef6122-7ec7-4246-86f1-45875122e8ec None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:26.193733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543a8f40-96dd-4ee7-9f69-d4103475d5d9 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:26.264608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7b82ba-0d71-4eff-9630-2e1726b5126e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:26.265137 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7b82ba-0d71-4eff-9630-2e1726b5126e None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:26.265423 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7b82ba-0d71-4eff-9630-2e1726b5126e None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:26.265694 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7b82ba-0d71-4eff-9630-2e1726b5126e None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:26.266372 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7b82ba-0d71-4eff-9630-2e1726b5126e None admin] RBAC: Adding query filter params (name=reader) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:26.266671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7b82ba-0d71-4eff-9630-2e1726b5126e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:26.268051 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7b82ba-0d71-4eff-9630-2e1726b5126e None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:26.295907 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 120/243] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:26 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:26.297612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7b82ba-0d71-4eff-9630-2e1726b5126e None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:26.304763 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:26.334980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:26.346052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cef6122-7ec7-4246-86f1-45875122e8ec None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:26.405545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6628981c-af18-48b5-8a0f-122f25d24012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:26.406660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6628981c-af18-48b5-8a0f-122f25d24012 None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:26.407238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6628981c-af18-48b5-8a0f-122f25d24012 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:26.407713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6628981c-af18-48b5-8a0f-122f25d24012 None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:26.408675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6628981c-af18-48b5-8a0f-122f25d24012 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:26.410515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6628981c-af18-48b5-8a0f-122f25d24012 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:26.429530 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7b82ba-0d71-4eff-9630-2e1726b5126e None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:26.469984 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a542a3b1-c6e2-4d24-9372-1621697216ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:26.470640 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a542a3b1-c6e2-4d24-9372-1621697216ca None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:26.470967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a542a3b1-c6e2-4d24-9372-1621697216ca None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:26.471312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a542a3b1-c6e2-4d24-9372-1621697216ca None admin] PATH_INFO: `/v3/users/40cbd8a9da674f0e8b11d3946ca3a911` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:26.486454 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a542a3b1-c6e2-4d24-9372-1621697216ca None admin] RBAC: Authorizing `identity:get_user(user_id=40cbd8a9da674f0e8b11d3946ca3a911)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:26.487902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a542a3b1-c6e2-4d24-9372-1621697216ca None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:26.494557 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 121/244] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:26 2022] GET /identity/v3/users/40cbd8a9da674f0e8b11d3946ca3a911 => generated 297 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:26.494557 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a542a3b1-c6e2-4d24-9372-1621697216ca None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:26.500510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 124/245] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:46:26 2022] POST /identity/v3/users => generated 293 bytes in 362 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 00:46:26.501687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6628981c-af18-48b5-8a0f-122f25d24012 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:26.633257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:26.690385 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a542a3b1-c6e2-4d24-9372-1621697216ca None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:26.703149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6628981c-af18-48b5-8a0f-122f25d24012 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:26.736926 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7da10a-d9b1-4dac-8cea-fcc72a946971 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:26.737981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7da10a-d9b1-4dac-8cea-fcc72a946971 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:26.738476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7da10a-d9b1-4dac-8cea-fcc72a946971 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:26.739296 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7da10a-d9b1-4dac-8cea-fcc72a946971 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:26.742220 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d15655-3433-4914-9f93-6d2c22df62b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:26.742836 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d15655-3433-4914-9f93-6d2c22df62b4 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:26.743139 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d15655-3433-4914-9f93-6d2c22df62b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:26.743558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d15655-3433-4914-9f93-6d2c22df62b4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:26.752607 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7da10a-d9b1-4dac-8cea-fcc72a946971 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:26.756924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:26.760016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-0f7da10a-d9b1-4dac-8cea-fcc72a946971 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:26.760016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:26.760016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:26.760016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:26.760016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:26.760016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:26.760016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:26.760016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:26.760016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:26.760016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:26.760016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:26.760016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:26.760016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:26.760016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:26.760016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:26.760016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:26.760867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:26.760867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:26.760867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:26.760867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:26.760867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:26.760867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:26.760867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:26.760867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:26.760867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:26.760867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:26.760867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:26.760867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:26.760867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:26.760867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:26.760867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:26.760867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:26.761676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:26.761676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:26.761676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:26.761676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:26.761676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:26.761676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:26.761676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:26.761676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:26.761676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:26.761676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:26.761676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:26.761676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:26.764020 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 122/246] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:26.769816 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7da10a-d9b1-4dac-8cea-fcc72a946971 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:26.818666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d15655-3433-4914-9f93-6d2c22df62b4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:26.820127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d15655-3433-4914-9f93-6d2c22df62b4 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:26.888409 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7da10a-d9b1-4dac-8cea-fcc72a946971 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:26.920623 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:26.925929 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 125/247] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:26.932842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d15655-3433-4914-9f93-6d2c22df62b4 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:26.936636 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898d2106-5037-46cd-9b18-9531d9b20d96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:26.937694 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898d2106-5037-46cd-9b18-9531d9b20d96 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:26.938160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898d2106-5037-46cd-9b18-9531d9b20d96 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:26.938609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898d2106-5037-46cd-9b18-9531d9b20d96 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:26.939466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898d2106-5037-46cd-9b18-9531d9b20d96 None admin] RBAC: Adding query filter params (name=member) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:26.939943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898d2106-5037-46cd-9b18-9531d9b20d96 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:26.942382 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898d2106-5037-46cd-9b18-9531d9b20d96 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:26.952119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 123/248] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:26.958671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898d2106-5037-46cd-9b18-9531d9b20d96 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:26.981871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:27.044540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d15655-3433-4914-9f93-6d2c22df62b4 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:27.071193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898d2106-5037-46cd-9b18-9531d9b20d96 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:27.081766 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ee21bb-5dbf-486d-b73a-70292cca5b8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:27.082967 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ee21bb-5dbf-486d-b73a-70292cca5b8d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:27.083278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ee21bb-5dbf-486d-b73a-70292cca5b8d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:27.083852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ee21bb-5dbf-486d-b73a-70292cca5b8d None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:27.085190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ee21bb-5dbf-486d-b73a-70292cca5b8d None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:27.086650 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ee21bb-5dbf-486d-b73a-70292cca5b8d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:27.090166 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 126/249] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:26 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:27.095980 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ee21bb-5dbf-486d-b73a-70292cca5b8d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:27.116531 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23006845-ccc8-4ab4-b854-16996ed035b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:27.117315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23006845-ccc8-4ab4-b854-16996ed035b5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:27.117652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23006845-ccc8-4ab4-b854-16996ed035b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:27.117934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23006845-ccc8-4ab4-b854-16996ed035b5 None admin] PATH_INFO: `/v3/users/9d0cf27ac9334ddfa7382dc6967ada80` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:27.119171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23006845-ccc8-4ab4-b854-16996ed035b5 None admin] RBAC: Authorizing `identity:get_user(user_id=9d0cf27ac9334ddfa7382dc6967ada80)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:27.120643 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23006845-ccc8-4ab4-b854-16996ed035b5 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:27.122924 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 124/250] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:26 2022] GET /identity/v3/users/9d0cf27ac9334ddfa7382dc6967ada80 => generated 297 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:27.128504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23006845-ccc8-4ab4-b854-16996ed035b5 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:27.194752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:27.209684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ee21bb-5dbf-486d-b73a-70292cca5b8d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:27.221006 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:27.240604 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23006845-ccc8-4ab4-b854-16996ed035b5 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:27.247580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1a01d2-ef85-4702-a9ca-9d6862aeb5e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:27.248461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1a01d2-ef85-4702-a9ca-9d6862aeb5e5 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:27.248924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1a01d2-ef85-4702-a9ca-9d6862aeb5e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:27.249354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1a01d2-ef85-4702-a9ca-9d6862aeb5e5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:27.250230 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1a01d2-ef85-4702-a9ca-9d6862aeb5e5 None admin] RBAC: Adding query filter params (role.id=6b0968db45c64a66b001a39d40579011, scope.project.id=d737f411ca514fb4aa93ee11d55926ae, user.id=40cbd8a9da674f0e8b11d3946ca3a911) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:27.256385 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1a01d2-ef85-4702-a9ca-9d6862aeb5e5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:27.256385 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1a01d2-ef85-4702-a9ca-9d6862aeb5e5 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:27.258932 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 127/251] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:27 2022] GET /identity/v3/role_assignments?user.id=40cbd8a9da674f0e8b11d3946ca3a911&scope.project.id=d737f411ca514fb4aa93ee11d55926ae&role.id=6b0968db45c64a66b001a39d40579011 => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:27.279833 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf6434c-2e2d-49ad-9f62-f4f190e22ece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:27.280489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf6434c-2e2d-49ad-9f62-f4f190e22ece None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:27.280768 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf6434c-2e2d-49ad-9f62-f4f190e22ece None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:27.281041 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf6434c-2e2d-49ad-9f62-f4f190e22ece None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:27.282297 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf6434c-2e2d-49ad-9f62-f4f190e22ece None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:27.284076 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf6434c-2e2d-49ad-9f62-f4f190e22ece None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:27.287773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 125/252] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:27 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:27.293749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1a01d2-ef85-4702-a9ca-9d6862aeb5e5 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:27.331231 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d3874d-ae4e-4252-ba53-1beaf1b2795c None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:27.331566 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d3874d-ae4e-4252-ba53-1beaf1b2795c None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:27.331787 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d3874d-ae4e-4252-ba53-1beaf1b2795c None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:27.334067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 126/253] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:27.340578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b082525f-ac56-4916-b25a-337a076202a2 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:27.340867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b082525f-ac56-4916-b25a-337a076202a2 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:27.341170 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b082525f-ac56-4916-b25a-337a076202a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:27.370116 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-b082525f-ac56-4916-b25a-337a076202a2 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:27.371370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b082525f-ac56-4916-b25a-337a076202a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:27.438081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1a01d2-ef85-4702-a9ca-9d6862aeb5e5 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:27.468106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 127/254] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:27.469234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2769ec-4892-42fd-8374-c968419d5707 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:27.469567 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2769ec-4892-42fd-8374-c968419d5707 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:27.469807 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2769ec-4892-42fd-8374-c968419d5707 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:27.471358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 128/255] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:27.477882 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7609a287-75e1-4951-b2d2-d8433f049418 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:27.478164 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7609a287-75e1-4951-b2d2-d8433f049418 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:27.478439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7609a287-75e1-4951-b2d2-d8433f049418 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:27.491046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf470d8-9ac9-4a44-a91f-199dceeda325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:27.491708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf470d8-9ac9-4a44-a91f-199dceeda325 None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:27.491990 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf470d8-9ac9-4a44-a91f-199dceeda325 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:27.492265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf470d8-9ac9-4a44-a91f-199dceeda325 None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae/users/9d0cf27ac9334ddfa7382dc6967ada80/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:27.494285 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf470d8-9ac9-4a44-a91f-199dceeda325 None admin] RBAC: Authorizing `identity:create_grant(project_id=d737f411ca514fb4aa93ee11d55926ae, user_id=9d0cf27ac9334ddfa7382dc6967ada80, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:27.495732 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf470d8-9ac9-4a44-a91f-199dceeda325 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:27.503320 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-7609a287-75e1-4951-b2d2-d8433f049418 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:27.509127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7609a287-75e1-4951-b2d2-d8433f049418 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:27.571682 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 128/256] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:46:27 2022] PUT /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae/users/9d0cf27ac9334ddfa7382dc6967ada80/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:46:27.597144 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:27.625412 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 129/257] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:46:27.883837 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf470d8-9ac9-4a44-a91f-199dceeda325 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:27.920516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:27.996946 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf470d8-9ac9-4a44-a91f-199dceeda325 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:28.004390 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7609a287-75e1-4951-b2d2-d8433f049418 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:28.042031 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb337765-e4d5-421d-a21a-d7019c92d3c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:28.042681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb337765-e4d5-421d-a21a-d7019c92d3c0 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:28.043167 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb337765-e4d5-421d-a21a-d7019c92d3c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:28.043606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb337765-e4d5-421d-a21a-d7019c92d3c0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:28.057678 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb337765-e4d5-421d-a21a-d7019c92d3c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:28.063580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-cb337765-e4d5-421d-a21a-d7019c92d3c0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:28.063580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:28.063580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:28.063580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:28.063580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:28.063580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:28.063580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:28.063580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:28.063580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:28.063580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:28.063580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:28.063580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:28.063580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:28.063580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:28.063580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:28.063580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:28.064671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:28.064671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:28.064671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:28.064671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:28.064671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:28.064671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:28.064671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:28.064671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:28.064671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:28.064671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:28.064671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:28.064671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:28.064671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:28.064671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:28.064671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:28.064671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:28.065702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:28.065702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:28.065702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:28.065702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:28.065702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:28.065702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:28.065702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:28.065702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:28.065702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:28.065702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:28.065702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:28.065702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:28.069174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 129/258] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:46:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:28.071310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb337765-e4d5-421d-a21a-d7019c92d3c0 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:28.115630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7609a287-75e1-4951-b2d2-d8433f049418 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:28.182560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da25a85a-0c69-4f3f-b90e-dffb959aadcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:28.183118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da25a85a-0c69-4f3f-b90e-dffb959aadcf None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:28.185107 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da25a85a-0c69-4f3f-b90e-dffb959aadcf None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:28.189858 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da25a85a-0c69-4f3f-b90e-dffb959aadcf None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:28.211126 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da25a85a-0c69-4f3f-b90e-dffb959aadcf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:28.218580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-da25a85a-0c69-4f3f-b90e-dffb959aadcf None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:28.218580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:28.218580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:28.218580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:28.218580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:28.218580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:28.218580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:28.218580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:28.218580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:28.218580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:28.218580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:28.218580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:28.218580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:28.218580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:28.218580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:28.218580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:28.219466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:28.219466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:28.219466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:28.219466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:28.219466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:28.219466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:28.219466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:28.219466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:28.219466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:28.219466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:28.219466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:28.219466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:28.219466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:28.219466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:28.219466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:28.219466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:28.220341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:28.220341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:28.220341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:28.220341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:28.220341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:28.220341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:28.220341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:28.220341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:28.220341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:28.220341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:28.220341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:28.220341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:28.220341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 130/259] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:46:28 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:46:28.234463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:28.241759 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb337765-e4d5-421d-a21a-d7019c92d3c0 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:28.246597 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da25a85a-0c69-4f3f-b90e-dffb959aadcf None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:28.327413 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bc5302-847c-4206-b753-d5523a4c1df4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:28.328075 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bc5302-847c-4206-b753-d5523a4c1df4 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:28.328354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bc5302-847c-4206-b753-d5523a4c1df4 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:28.328670 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bc5302-847c-4206-b753-d5523a4c1df4 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:28.333562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27bc5302-847c-4206-b753-d5523a4c1df4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:28.339158 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27bc5302-847c-4206-b753-d5523a4c1df4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:28.339158 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27bc5302-847c-4206-b753-d5523a4c1df4 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:28.352474 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 130/260] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:28.374491 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bc5302-847c-4206-b753-d5523a4c1df4 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:28.384405 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:28.472026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da25a85a-0c69-4f3f-b90e-dffb959aadcf None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:28.532553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a80874-144f-4701-927f-4295b64636db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:28.533562 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a80874-144f-4701-927f-4295b64636db None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:28.534045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a80874-144f-4701-927f-4295b64636db None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:28.534517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a80874-144f-4701-927f-4295b64636db None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:28.535409 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a80874-144f-4701-927f-4295b64636db None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:28.536462 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a80874-144f-4701-927f-4295b64636db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:28.539343 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a80874-144f-4701-927f-4295b64636db None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:28.565096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 131/261] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:46:28 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:28.567583 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a80874-144f-4701-927f-4295b64636db None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:28.595505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bc5302-847c-4206-b753-d5523a4c1df4 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:28.677841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef4a8c2-a770-4834-9bac-5b46c403fa8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:28.680174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef4a8c2-a770-4834-9bac-5b46c403fa8b None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:28.682031 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef4a8c2-a770-4834-9bac-5b46c403fa8b None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:28.682522 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef4a8c2-a770-4834-9bac-5b46c403fa8b None admin] PATH_INFO: `/v3/users/1d534fbaeec64a2bbaf523be4f6cf886` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:28.684810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef4a8c2-a770-4834-9bac-5b46c403fa8b None admin] RBAC: Authorizing `identity:get_user(user_id=1d534fbaeec64a2bbaf523be4f6cf886)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:28.687430 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef4a8c2-a770-4834-9bac-5b46c403fa8b None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:28.691476 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 131/262] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:28 2022] GET /identity/v3/users/1d534fbaeec64a2bbaf523be4f6cf886 => generated 283 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:28.697223 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef4a8c2-a770-4834-9bac-5b46c403fa8b None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:28.757558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:28.785421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a80874-144f-4701-927f-4295b64636db None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:28.807604 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:28.875125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a859739d-7ae5-4413-8203-f4c08847c5aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:28.875796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a859739d-7ae5-4413-8203-f4c08847c5aa None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:28.876078 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a859739d-7ae5-4413-8203-f4c08847c5aa None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:28.876356 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a859739d-7ae5-4413-8203-f4c08847c5aa None admin] PATH_INFO: `/v3/users/1d534fbaeec64a2bbaf523be4f6cf886` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:28.877919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a859739d-7ae5-4413-8203-f4c08847c5aa None admin] RBAC: Authorizing `identity:get_user(user_id=1d534fbaeec64a2bbaf523be4f6cf886)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:28.879914 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a859739d-7ae5-4413-8203-f4c08847c5aa None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:28.881645 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 132/263] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:28 2022] GET /identity/v3/users/1d534fbaeec64a2bbaf523be4f6cf886 => generated 283 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:28.888843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a859739d-7ae5-4413-8203-f4c08847c5aa None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:28.921815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef4a8c2-a770-4834-9bac-5b46c403fa8b None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:28.993472 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2ea0b4-c769-4ae6-bfd8-153f521fd940 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:28.994089 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2ea0b4-c769-4ae6-bfd8-153f521fd940 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:28.994370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2ea0b4-c769-4ae6-bfd8-153f521fd940 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:28.994640 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2ea0b4-c769-4ae6-bfd8-153f521fd940 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:29.067488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a859739d-7ae5-4413-8203-f4c08847c5aa None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:29.088087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2ea0b4-c769-4ae6-bfd8-153f521fd940 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:29.089610 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2ea0b4-c769-4ae6-bfd8-153f521fd940 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:29.142114 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:29.178660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36533db3-4766-4816-b788-375f03a8c1bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:29.179286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36533db3-4766-4816-b788-375f03a8c1bd None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:29.179562 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36533db3-4766-4816-b788-375f03a8c1bd None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:29.179834 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36533db3-4766-4816-b788-375f03a8c1bd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:29.199569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 132/264] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 503 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:29.209358 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2ea0b4-c769-4ae6-bfd8-153f521fd940 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:29.272561 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:29.309596 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36533db3-4766-4816-b788-375f03a8c1bd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:29.319070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36533db3-4766-4816-b788-375f03a8c1bd None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:29.382520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2ea0b4-c769-4ae6-bfd8-153f521fd940 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:29.425467 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:29.450506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304b70ef-29a8-47e7-b9eb-ee403a08aa88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:29.451163 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304b70ef-29a8-47e7-b9eb-ee403a08aa88 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:29.451449 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304b70ef-29a8-47e7-b9eb-ee403a08aa88 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:29.451726 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304b70ef-29a8-47e7-b9eb-ee403a08aa88 None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:29.455077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304b70ef-29a8-47e7-b9eb-ee403a08aa88 None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:29.456546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304b70ef-29a8-47e7-b9eb-ee403a08aa88 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:29.459321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 133/265] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:29 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:29.474469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304b70ef-29a8-47e7-b9eb-ee403a08aa88 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:29.495857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 133/266] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 607 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:29.502021 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36533db3-4766-4816-b788-375f03a8c1bd None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:29.662723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304b70ef-29a8-47e7-b9eb-ee403a08aa88 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:29.709794 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12228f7-1587-4294-8196-34641aeec4d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:29.710416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12228f7-1587-4294-8196-34641aeec4d8 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:29.710701 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12228f7-1587-4294-8196-34641aeec4d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:29.710978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12228f7-1587-4294-8196-34641aeec4d8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:29.711800 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12228f7-1587-4294-8196-34641aeec4d8 None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=d737f411ca514fb4aa93ee11d55926ae, user.id=1d534fbaeec64a2bbaf523be4f6cf886) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:29.712121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12228f7-1587-4294-8196-34641aeec4d8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:29.713606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12228f7-1587-4294-8196-34641aeec4d8 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:29.721567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 134/267] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:29 2022] GET /identity/v3/role_assignments?user.id=1d534fbaeec64a2bbaf523be4f6cf886&scope.project.id=d737f411ca514fb4aa93ee11d55926ae&role.id=19c006412bec45dc8febd7cc86b9678f => generated 608 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:29.722154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:29.760130 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36533db3-4766-4816-b788-375f03a8c1bd None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:29.812744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7486769c-28af-490f-aead-c0867758aa91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:29.813422 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7486769c-28af-490f-aead-c0867758aa91 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:29.813701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7486769c-28af-490f-aead-c0867758aa91 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:29.813975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7486769c-28af-490f-aead-c0867758aa91 None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:29.818120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7486769c-28af-490f-aead-c0867758aa91 None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:29.819626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7486769c-28af-490f-aead-c0867758aa91 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:29.821469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 134/268] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:29 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:29.827931 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12228f7-1587-4294-8196-34641aeec4d8 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:29.922542 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:29.987283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12228f7-1587-4294-8196-34641aeec4d8 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:29.994044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae8c8e5-2a1d-4762-bd22-38b7d30843a1 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:29.994435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae8c8e5-2a1d-4762-bd22-38b7d30843a1 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:29.994856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae8c8e5-2a1d-4762-bd22-38b7d30843a1 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:29.997013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 135/269] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:30.002725 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68f839c-fbe5-4ac3-94b7-37e65ade672b None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:30.003031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68f839c-fbe5-4ac3-94b7-37e65ade672b None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:30.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68f839c-fbe5-4ac3-94b7-37e65ade672b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:30.032484 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-b68f839c-fbe5-4ac3-94b7-37e65ade672b None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:30.033747 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b68f839c-fbe5-4ac3-94b7-37e65ade672b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:30.034597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15f164c-2893-46ae-8456-ccaadfa45781 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:30.035262 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15f164c-2893-46ae-8456-ccaadfa45781 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:30.036213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15f164c-2893-46ae-8456-ccaadfa45781 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:30.036490 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15f164c-2893-46ae-8456-ccaadfa45781 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:30.037257 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15f164c-2893-46ae-8456-ccaadfa45781 None admin] RBAC: Adding query filter params (role.id=c599b217579e42a49c657c642ea0648f, scope.project.id=d737f411ca514fb4aa93ee11d55926ae, user.id=1d534fbaeec64a2bbaf523be4f6cf886) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:30.037564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15f164c-2893-46ae-8456-ccaadfa45781 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:30.038976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15f164c-2893-46ae-8456-ccaadfa45781 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:30.051259 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 135/270] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:29 2022] GET /identity/v3/role_assignments?user.id=1d534fbaeec64a2bbaf523be4f6cf886&scope.project.id=d737f411ca514fb4aa93ee11d55926ae&role.id=c599b217579e42a49c657c642ea0648f => generated 608 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:30.112381 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 136/271] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:46:30.165782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbccfb3-94e8-46eb-a184-acb2e9ea667e None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:30.166086 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbccfb3-94e8-46eb-a184-acb2e9ea667e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:30.166341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbccfb3-94e8-46eb-a184-acb2e9ea667e None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:30.167986 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 136/272] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:30.175093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cd2c27-ee6f-4391-962a-506e3b05698e None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:30.175580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cd2c27-ee6f-4391-962a-506e3b05698e None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:30.175994 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cd2c27-ee6f-4391-962a-506e3b05698e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:30.198886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-12cd2c27-ee6f-4391-962a-506e3b05698e None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:30.200442 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-12cd2c27-ee6f-4391-962a-506e3b05698e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:30.216317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff32ad3-e5e6-438b-b89b-1ad8e979dd84 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:30.216833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff32ad3-e5e6-438b-b89b-1ad8e979dd84 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:30.217275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff32ad3-e5e6-438b-b89b-1ad8e979dd84 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:30.219008 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 137/273] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:46:30.226349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ac22bd-1242-44cd-83ad-703939612b8c None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:30.226950 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ac22bd-1242-44cd-83ad-703939612b8c None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:30.227419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ac22bd-1242-44cd-83ad-703939612b8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:30.255876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-26ac22bd-1242-44cd-83ad-703939612b8c None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:30.257598 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-26ac22bd-1242-44cd-83ad-703939612b8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:30.314480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 137/274] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:30.351166 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 138/275] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:46:30.499398 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cd2c27-ee6f-4391-962a-506e3b05698e None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:30.503905 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:30.606708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48de9bb-a643-4494-abef-0c658d3c2b88 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:30.607059 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48de9bb-a643-4494-abef-0c658d3c2b88 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:30.607307 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48de9bb-a643-4494-abef-0c658d3c2b88 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:30.608934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 139/276] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:46:30.609699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48de9bb-a643-4494-abef-0c658d3c2b88 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:30.630453 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cd2c27-ee6f-4391-962a-506e3b05698e None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:30.666596 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:30.667110 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41cfed7-60c5-4eaf-a6fd-f66438206cbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:30.667722 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41cfed7-60c5-4eaf-a6fd-f66438206cbc None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:30.668001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41cfed7-60c5-4eaf-a6fd-f66438206cbc None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:30.668274 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41cfed7-60c5-4eaf-a6fd-f66438206cbc None admin] PATH_INFO: `/v3/roles/member` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:30.679255 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41cfed7-60c5-4eaf-a6fd-f66438206cbc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:30.684230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-a41cfed7-60c5-4eaf-a6fd-f66438206cbc None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:30.684230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:30.684230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:30.684230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:30.684230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:30.684230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:30.684230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:30.684230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:30.684230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:30.684230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:30.684230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:30.684230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:30.684230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:30.684230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:30.684230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:30.684230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:30.685209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:30.685209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:30.685209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:30.685209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:30.685209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:30.685209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:30.685209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:30.685209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:30.685209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:30.685209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:30.685209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:30.685209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:30.685209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:30.685209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:30.685209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:30.685209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:30.686140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:30.686140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:30.686140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:30.686140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:30.686140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:30.686140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:30.686140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:30.686140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:30.686140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:30.686140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:30.686140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:30.686140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:30.686140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 138/277] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:30.686913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75457e39-b20b-47fd-bb0d-8b1fb41895d4 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:30.687099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75457e39-b20b-47fd-bb0d-8b1fb41895d4 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:30.687346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75457e39-b20b-47fd-bb0d-8b1fb41895d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:30.705050 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-75457e39-b20b-47fd-bb0d-8b1fb41895d4 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:30.706266 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-75457e39-b20b-47fd-bb0d-8b1fb41895d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:30.719572 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48de9bb-a643-4494-abef-0c658d3c2b88 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:30.753360 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19574354-8f29-4c4a-bb1d-b54591da7e68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:30.753985 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19574354-8f29-4c4a-bb1d-b54591da7e68 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:30.754308 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19574354-8f29-4c4a-bb1d-b54591da7e68 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:30.754631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19574354-8f29-4c4a-bb1d-b54591da7e68 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:30.755723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19574354-8f29-4c4a-bb1d-b54591da7e68 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:30.757183 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19574354-8f29-4c4a-bb1d-b54591da7e68 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:30.758731 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 140/278] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:46:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:30.761265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19574354-8f29-4c4a-bb1d-b54591da7e68 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:30.767132 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:30.781126 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 139/279] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:30.781126 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75457e39-b20b-47fd-bb0d-8b1fb41895d4 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:30.864132 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19574354-8f29-4c4a-bb1d-b54591da7e68 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:30.882220 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:30.889443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75457e39-b20b-47fd-bb0d-8b1fb41895d4 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:30.900060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fbdd5c-df85-431d-b766-23492050a06e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:30.900691 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fbdd5c-df85-431d-b766-23492050a06e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:30.900971 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fbdd5c-df85-431d-b766-23492050a06e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:30.901277 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fbdd5c-df85-431d-b766-23492050a06e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:30.912611 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fbdd5c-df85-431d-b766-23492050a06e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:30.918004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-04fbdd5c-df85-431d-b766-23492050a06e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:30.918004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:30.918004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:30.918004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:30.918004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:30.918004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:30.918004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:30.918004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:30.918004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:30.918004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:30.918004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:30.918004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:30.918004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:30.918004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:30.918004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:30.918004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:30.919113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:30.919113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:30.919113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:30.919113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:30.919113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:30.919113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:30.919113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:30.919113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:30.919113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:30.919113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:30.919113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:30.919113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:30.919113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:30.919113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:30.919113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:30.919113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:30.921382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:30.921382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:30.921382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:30.921382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:30.921382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:30.921382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:30.921382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:30.921382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:30.921382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:30.921382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:30.921382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:30.921382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:30.921382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 141/280] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:30 2022] GET /identity/v3/roles/reader => generated 84 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:30.921382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fbdd5c-df85-431d-b766-23492050a06e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:30.933058 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5c407a-3f35-4ae0-a301-046ca36af6e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:30.933800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5c407a-3f35-4ae0-a301-046ca36af6e5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:30.934080 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5c407a-3f35-4ae0-a301-046ca36af6e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:30.934353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5c407a-3f35-4ae0-a301-046ca36af6e5 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:30.935020 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5c407a-3f35-4ae0-a301-046ca36af6e5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:30.935440 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5c407a-3f35-4ae0-a301-046ca36af6e5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:30.936820 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5c407a-3f35-4ae0-a301-046ca36af6e5 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:30.946097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 140/281] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:30 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:30.946912 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5c407a-3f35-4ae0-a301-046ca36af6e5 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:31.007957 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:31.036728 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fbdd5c-df85-431d-b766-23492050a06e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:31.053214 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5c407a-3f35-4ae0-a301-046ca36af6e5 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:31.070819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af800d7-9b17-4100-a51e-673728d5a56b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:31.071431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af800d7-9b17-4100-a51e-673728d5a56b None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:31.071714 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af800d7-9b17-4100-a51e-673728d5a56b None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:31.071989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af800d7-9b17-4100-a51e-673728d5a56b None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:31.072641 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af800d7-9b17-4100-a51e-673728d5a56b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:31.074099 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af800d7-9b17-4100-a51e-673728d5a56b None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:31.087629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a035e904-96b3-491d-bef3-51d68f6ca173 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:31.088234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a035e904-96b3-491d-bef3-51d68f6ca173 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:31.088516 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a035e904-96b3-491d-bef3-51d68f6ca173 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:31.088792 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a035e904-96b3-491d-bef3-51d68f6ca173 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:31.089494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a035e904-96b3-491d-bef3-51d68f6ca173 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:31.089798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a035e904-96b3-491d-bef3-51d68f6ca173 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:31.091162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a035e904-96b3-491d-bef3-51d68f6ca173 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:31.100162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 141/282] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:30 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:31.100948 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a035e904-96b3-491d-bef3-51d68f6ca173 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:31.192842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 142/283] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:46:30 2022] POST /identity/v3/users => generated 293 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:46:31.193834 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af800d7-9b17-4100-a51e-673728d5a56b None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:31.200230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a035e904-96b3-491d-bef3-51d68f6ca173 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:31.220074 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:31.238715 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55243846-c379-40dd-a4da-160dcde1264c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:31.239376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55243846-c379-40dd-a4da-160dcde1264c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:31.239771 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55243846-c379-40dd-a4da-160dcde1264c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:31.240056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55243846-c379-40dd-a4da-160dcde1264c None admin] PATH_INFO: `/v3/users/f54e8c9978684b00bc21d66dfa3397d9` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:31.247592 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:31.255742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55243846-c379-40dd-a4da-160dcde1264c None admin] RBAC: Authorizing `identity:get_user(user_id=f54e8c9978684b00bc21d66dfa3397d9)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:31.257287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55243846-c379-40dd-a4da-160dcde1264c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:31.259110 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 142/284] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:31 2022] GET /identity/v3/users/f54e8c9978684b00bc21d66dfa3397d9 => generated 293 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:31.259868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55243846-c379-40dd-a4da-160dcde1264c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:31.334548 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af800d7-9b17-4100-a51e-673728d5a56b None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:31.376494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55243846-c379-40dd-a4da-160dcde1264c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:31.379111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe65df9-e541-416a-982e-ea97b6fae440 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:31.380537 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe65df9-e541-416a-982e-ea97b6fae440 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:31.380817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe65df9-e541-416a-982e-ea97b6fae440 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:31.381104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe65df9-e541-416a-982e-ea97b6fae440 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:31.391614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe65df9-e541-416a-982e-ea97b6fae440 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:31.397524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-ebe65df9-e541-416a-982e-ea97b6fae440 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:31.397524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:31.397524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:31.397524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:31.397524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:31.397524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:31.397524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:31.397524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:31.397524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:31.397524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:31.397524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:31.397524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:31.397524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:31.397524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:31.397524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:31.397524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:31.398401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:31.398401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:31.398401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:31.398401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:31.398401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:31.398401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:31.398401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:31.398401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:31.398401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:31.398401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:31.398401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:31.398401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:31.398401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:31.398401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:31.398401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:31.398401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:31.399602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:31.399602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:31.399602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:31.399602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:31.399602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:31.399602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:31.399602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:31.399602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:31.399602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:31.399602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:31.399602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:31.399602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:31.399602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 143/285] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:31.401950 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe65df9-e541-416a-982e-ea97b6fae440 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:31.423527 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16972e42-e595-4af8-814a-83c9fa5cc607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:31.424546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16972e42-e595-4af8-814a-83c9fa5cc607 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:31.424546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16972e42-e595-4af8-814a-83c9fa5cc607 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:31.424811 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16972e42-e595-4af8-814a-83c9fa5cc607 None admin] PATH_INFO: `/v3/users/40cbd8a9da674f0e8b11d3946ca3a911` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:31.429849 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16972e42-e595-4af8-814a-83c9fa5cc607 None admin] RBAC: Authorizing `identity:get_user(user_id=40cbd8a9da674f0e8b11d3946ca3a911)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:31.432365 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16972e42-e595-4af8-814a-83c9fa5cc607 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:31.434330 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 143/286] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:31 2022] GET /identity/v3/users/40cbd8a9da674f0e8b11d3946ca3a911 => generated 297 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:31.435322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16972e42-e595-4af8-814a-83c9fa5cc607 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:31.494986 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:31.531757 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:31.539482 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe65df9-e541-416a-982e-ea97b6fae440 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:31.568522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16972e42-e595-4af8-814a-83c9fa5cc607 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:31.587992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c4fad6-0964-4b18-9659-00f022fbe54f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:31.588849 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c4fad6-0964-4b18-9659-00f022fbe54f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:31.589538 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c4fad6-0964-4b18-9659-00f022fbe54f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:31.589817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c4fad6-0964-4b18-9659-00f022fbe54f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:31.590554 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c4fad6-0964-4b18-9659-00f022fbe54f None admin] RBAC: Adding query filter params (role.id=4bfd6d0544ee4a9e933538e78135a12c, user.id=f54e8c9978684b00bc21d66dfa3397d9, scope.system=all) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:31.590862 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c4fad6-0964-4b18-9659-00f022fbe54f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:31.592858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c4fad6-0964-4b18-9659-00f022fbe54f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:31.600483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 144/287] 10.222.0.42 () {56 vars in 1308 bytes} [Sun Sep 4 00:46:31 2022] GET /identity/v3/role_assignments?user.id=f54e8c9978684b00bc21d66dfa3397d9&scope.system=all&role.id=4bfd6d0544ee4a9e933538e78135a12c => generated 226 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:31.601723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c4fad6-0964-4b18-9659-00f022fbe54f None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:31.608327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6644888e-b2ce-4346-8b5c-606d9e52a85c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:31.609010 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6644888e-b2ce-4346-8b5c-606d9e52a85c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:31.609342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6644888e-b2ce-4346-8b5c-606d9e52a85c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:31.609670 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6644888e-b2ce-4346-8b5c-606d9e52a85c None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:31.610369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6644888e-b2ce-4346-8b5c-606d9e52a85c None admin] RBAC: Adding query filter params (name=member) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:31.610736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6644888e-b2ce-4346-8b5c-606d9e52a85c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:31.612302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6644888e-b2ce-4346-8b5c-606d9e52a85c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:31.623453 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 144/288] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:31.630726 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6644888e-b2ce-4346-8b5c-606d9e52a85c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:31.794675 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6644888e-b2ce-4346-8b5c-606d9e52a85c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:31.795417 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c4fad6-0964-4b18-9659-00f022fbe54f None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:31.802088 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:31.831592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcda8a70-99e4-4388-8dfd-756282035a90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:31.833236 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcda8a70-99e4-4388-8dfd-756282035a90 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:31.833880 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcda8a70-99e4-4388-8dfd-756282035a90 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:31.834346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcda8a70-99e4-4388-8dfd-756282035a90 None admin] PATH_INFO: `/v3/users/9d0cf27ac9334ddfa7382dc6967ada80` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:31.835844 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcda8a70-99e4-4388-8dfd-756282035a90 None admin] RBAC: Authorizing `identity:get_user(user_id=9d0cf27ac9334ddfa7382dc6967ada80)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:31.837792 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcda8a70-99e4-4388-8dfd-756282035a90 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:31.839957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ef0a03-370e-4af5-9ff7-18689972db23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:31.840234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 145/289] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:31 2022] GET /identity/v3/users/9d0cf27ac9334ddfa7382dc6967ada80 => generated 297 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:31.840595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ef0a03-370e-4af5-9ff7-18689972db23 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:31.840887 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ef0a03-370e-4af5-9ff7-18689972db23 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:31.841356 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ef0a03-370e-4af5-9ff7-18689972db23 None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:31.842354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ef0a03-370e-4af5-9ff7-18689972db23 None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:31.843784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ef0a03-370e-4af5-9ff7-18689972db23 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:31.845758 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcda8a70-99e4-4388-8dfd-756282035a90 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:31.849527 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 145/290] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:31 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:31.853965 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ef0a03-370e-4af5-9ff7-18689972db23 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:31.897226 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:32.021569 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcda8a70-99e4-4388-8dfd-756282035a90 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:32.042916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ef0a03-370e-4af5-9ff7-18689972db23 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:32.086520 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2eb550-ea8a-4fad-a72a-1f66050eb091 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:32.087130 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2eb550-ea8a-4fad-a72a-1f66050eb091 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:32.087402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2eb550-ea8a-4fad-a72a-1f66050eb091 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:32.087668 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2eb550-ea8a-4fad-a72a-1f66050eb091 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:32.106478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02df8ff-38f2-4103-822b-0bb8fba6adb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:32.107505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02df8ff-38f2-4103-822b-0bb8fba6adb7 None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:32.108017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02df8ff-38f2-4103-822b-0bb8fba6adb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:32.108494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02df8ff-38f2-4103-822b-0bb8fba6adb7 None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae/users/40cbd8a9da674f0e8b11d3946ca3a911/roles/6b0968db45c64a66b001a39d40579011` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:32.117858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02df8ff-38f2-4103-822b-0bb8fba6adb7 None admin] RBAC: Authorizing `identity:create_grant(project_id=d737f411ca514fb4aa93ee11d55926ae, user_id=40cbd8a9da674f0e8b11d3946ca3a911, role_id=6b0968db45c64a66b001a39d40579011)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:32.119555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02df8ff-38f2-4103-822b-0bb8fba6adb7 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:32.136167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:32.161355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2eb550-ea8a-4fad-a72a-1f66050eb091 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:32.163052 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2eb550-ea8a-4fad-a72a-1f66050eb091 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:32.251265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 146/291] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:46:31 2022] PUT /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae/users/40cbd8a9da674f0e8b11d3946ca3a911/roles/6b0968db45c64a66b001a39d40579011 => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:46:32.270118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 146/292] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:32.275976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02df8ff-38f2-4103-822b-0bb8fba6adb7 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:32.445890 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:32.463847 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02df8ff-38f2-4103-822b-0bb8fba6adb7 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:32.514026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97584d0d-4772-416f-a87b-8cd00a8ca5a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:32.514681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97584d0d-4772-416f-a87b-8cd00a8ca5a4 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:32.514963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97584d0d-4772-416f-a87b-8cd00a8ca5a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:32.515234 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97584d0d-4772-416f-a87b-8cd00a8ca5a4 None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:32.516433 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97584d0d-4772-416f-a87b-8cd00a8ca5a4 None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:32.517892 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97584d0d-4772-416f-a87b-8cd00a8ca5a4 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:32.519603 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 147/293] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:32 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:32.526309 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2eb550-ea8a-4fad-a72a-1f66050eb091 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:32.610497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:32.636918 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2eb550-ea8a-4fad-a72a-1f66050eb091 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:32.674223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a759dcf2-1148-4cff-841f-abed2a08a555 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:32.674832 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a759dcf2-1148-4cff-841f-abed2a08a555 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:32.675108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a759dcf2-1148-4cff-841f-abed2a08a555 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:32.675382 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a759dcf2-1148-4cff-841f-abed2a08a555 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:32.676677 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a759dcf2-1148-4cff-841f-abed2a08a555 None admin] RBAC: Adding query filter params (role.id=4bfd6d0544ee4a9e933538e78135a12c, scope.project.id=d737f411ca514fb4aa93ee11d55926ae, user.id=9d0cf27ac9334ddfa7382dc6967ada80) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:32.676988 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a759dcf2-1148-4cff-841f-abed2a08a555 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:32.678512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a759dcf2-1148-4cff-841f-abed2a08a555 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:32.684373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 147/294] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:32 2022] GET /identity/v3/role_assignments?user.id=9d0cf27ac9334ddfa7382dc6967ada80&scope.project.id=d737f411ca514fb4aa93ee11d55926ae&role.id=4bfd6d0544ee4a9e933538e78135a12c => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:33.444214 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59641a82-8aa6-45a0-a91c-163c8b5e3c1a None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:33.445315 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59641a82-8aa6-45a0-a91c-163c8b5e3c1a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:33.445806 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59641a82-8aa6-45a0-a91c-163c8b5e3c1a None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:33.447376 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 148/295] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:33.452927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cce3910-8105-47bc-8a66-be7541a764ca None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:33.453488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cce3910-8105-47bc-8a66-be7541a764ca None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:33.454017 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cce3910-8105-47bc-8a66-be7541a764ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:33.479257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-6cce3910-8105-47bc-8a66-be7541a764ca None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:33.480771 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-6cce3910-8105-47bc-8a66-be7541a764ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:33.552540 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 148/296] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:33.613175 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1592f077-de75-405e-87bb-1a52d9b220c8 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:33.614538 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1592f077-de75-405e-87bb-1a52d9b220c8 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:33.615107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1592f077-de75-405e-87bb-1a52d9b220c8 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:33.616918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 149/297] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:33 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:33.624344 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9a4bb6-eebc-49eb-9184-4a97c5adf1ef None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:33.625175 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9a4bb6-eebc-49eb-9184-4a97c5adf1ef None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:33.625679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9a4bb6-eebc-49eb-9184-4a97c5adf1ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:33.645782 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-ee9a4bb6-eebc-49eb-9184-4a97c5adf1ef None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:33.647318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ee9a4bb6-eebc-49eb-9184-4a97c5adf1ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:33.713610 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:33.719012 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 149/298] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:33.840239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1592f077-de75-405e-87bb-1a52d9b220c8 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:33.865642 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafb5d16-bad0-44b9-99c4-3d444ade04af None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:33.866258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafb5d16-bad0-44b9-99c4-3d444ade04af None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:33.866866 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafb5d16-bad0-44b9-99c4-3d444ade04af None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:33.869301 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 150/299] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:33 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:46:33.876010 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850ef3f0-3b76-4e4a-843c-97e3fd8d921f None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:33.876564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850ef3f0-3b76-4e4a-843c-97e3fd8d921f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:33.877055 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850ef3f0-3b76-4e4a-843c-97e3fd8d921f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:33.897455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-850ef3f0-3b76-4e4a-843c-97e3fd8d921f None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:33.898944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-850ef3f0-3b76-4e4a-843c-97e3fd8d921f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:33.967543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1592f077-de75-405e-87bb-1a52d9b220c8 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:33.988323 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 151/300] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:46:34.003052 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850ef3f0-3b76-4e4a-843c-97e3fd8d921f None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:34.009737 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a5e354-f654-4ce6-b8ef-748c17e92829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:34.010376 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a5e354-f654-4ce6-b8ef-748c17e92829 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:34.010654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a5e354-f654-4ce6-b8ef-748c17e92829 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:34.010923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a5e354-f654-4ce6-b8ef-748c17e92829 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:34.025615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a5e354-f654-4ce6-b8ef-748c17e92829 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:34.032665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-f9a5e354-f654-4ce6-b8ef-748c17e92829 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:34.032665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:34.032665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:34.032665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:34.032665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:34.032665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:34.032665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:34.032665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:34.032665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:34.032665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:34.032665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:34.032665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:34.032665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:34.032665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:34.032665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:34.032665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:34.033584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:34.033584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:34.033584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:34.033584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:34.033584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:34.033584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:34.033584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:34.033584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:34.033584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:34.033584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:34.033584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:34.033584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:34.033584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:34.033584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:34.033584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:34.033584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:34.034353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:34.034353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:34.034353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:34.034353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:34.034353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:34.034353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:34.034353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:34.034353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:34.034353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:34.034353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:34.034353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:34.034353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:34.034961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 150/301] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:33 2022] GET /identity/v3/roles/reader => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:34.041963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a5e354-f654-4ce6-b8ef-748c17e92829 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:34.059919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:34.133016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850ef3f0-3b76-4e4a-843c-97e3fd8d921f None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:34.156080 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a5e354-f654-4ce6-b8ef-748c17e92829 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:34.166422 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:34.176482 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd913d3-5132-41d9-a3f8-0496beb5d2c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:34.177843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd913d3-5132-41d9-a3f8-0496beb5d2c4 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:34.178294 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd913d3-5132-41d9-a3f8-0496beb5d2c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:34.178980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd913d3-5132-41d9-a3f8-0496beb5d2c4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:34.180754 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd913d3-5132-41d9-a3f8-0496beb5d2c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:34.183101 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd913d3-5132-41d9-a3f8-0496beb5d2c4 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:34.185623 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 152/302] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:46:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:34.190850 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd913d3-5132-41d9-a3f8-0496beb5d2c4 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:34.203573 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed663b4-135d-4619-9a0c-953b3886245d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:34.204449 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed663b4-135d-4619-9a0c-953b3886245d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:34.204940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed663b4-135d-4619-9a0c-953b3886245d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:34.205420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed663b4-135d-4619-9a0c-953b3886245d None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:34.206280 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed663b4-135d-4619-9a0c-953b3886245d None admin] RBAC: Adding query filter params (name=reader) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:34.206766 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed663b4-135d-4619-9a0c-953b3886245d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:34.208902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed663b4-135d-4619-9a0c-953b3886245d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:34.223340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 151/303] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:34 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:34.224950 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed663b4-135d-4619-9a0c-953b3886245d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:34.305162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd913d3-5132-41d9-a3f8-0496beb5d2c4 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:34.325405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:34.341883 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6a29ca-1808-4d9a-8c71-889bcd7da35e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:34.342574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6a29ca-1808-4d9a-8c71-889bcd7da35e None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:34.342771 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6a29ca-1808-4d9a-8c71-889bcd7da35e None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:34.343043 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6a29ca-1808-4d9a-8c71-889bcd7da35e None admin] PATH_INFO: `/v3/groups` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:34.343681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6a29ca-1808-4d9a-8c71-889bcd7da35e None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:34.345134 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6a29ca-1808-4d9a-8c71-889bcd7da35e None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:34.348751 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed663b4-135d-4619-9a0c-953b3886245d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:34.383203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb944e48-56b7-4eb0-913f-b764ed90f814 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:34.383905 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb944e48-56b7-4eb0-913f-b764ed90f814 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:34.384188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb944e48-56b7-4eb0-913f-b764ed90f814 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:34.384456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb944e48-56b7-4eb0-913f-b764ed90f814 None admin] PATH_INFO: `/v3/users/161ef36cfa2f4420ad2be2a75ce51f01` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:34.398086 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb944e48-56b7-4eb0-913f-b764ed90f814 None admin] RBAC: Authorizing `identity:get_user(user_id=161ef36cfa2f4420ad2be2a75ce51f01)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:34.399517 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb944e48-56b7-4eb0-913f-b764ed90f814 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:34.401251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 152/304] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:34 2022] GET /identity/v3/users/161ef36cfa2f4420ad2be2a75ce51f01 => generated 293 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:34.406389 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb944e48-56b7-4eb0-913f-b764ed90f814 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:34.435028 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 153/305] 10.222.0.42 () {60 vars in 1134 bytes} [Sun Sep 4 00:46:34 2022] POST /identity/v3/groups => generated 229 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 00:46:34.445110 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6a29ca-1808-4d9a-8c71-889bcd7da35e None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:34.529290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb944e48-56b7-4eb0-913f-b764ed90f814 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:34.535734 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:34.587786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6a29ca-1808-4d9a-8c71-889bcd7da35e None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:34.591215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cdfc35-1d70-432f-96d6-6020c3af3ba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:34.591215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cdfc35-1d70-432f-96d6-6020c3af3ba6 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:34.591548 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cdfc35-1d70-432f-96d6-6020c3af3ba6 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:34.591774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cdfc35-1d70-432f-96d6-6020c3af3ba6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:34.595669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:34.611342 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cdfc35-1d70-432f-96d6-6020c3af3ba6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:34.616185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-21cdfc35-1d70-432f-96d6-6020c3af3ba6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:34.616185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:34.616185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:34.616185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:34.616185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:34.616185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:34.616185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:34.616185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:34.616185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:34.616185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:34.616185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:34.616185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:34.616185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:34.616185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:34.616185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:34.616185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:34.617109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:34.617109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:34.617109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:34.617109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:34.617109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:34.617109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:34.617109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:34.617109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:34.617109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:34.617109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:34.617109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:34.617109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:34.617109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:34.617109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:34.617109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:34.617109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:34.617897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:34.617897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:34.617897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:34.617897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:34.617897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:34.617897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:34.617897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:34.617897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:34.617897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:34.617897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:34.617897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:34.617897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:34.617897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 153/306] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:34 2022] GET /identity/v3/roles/member => generated 84 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:34.620123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d785995-0b74-48c5-9354-a524e7f382db None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:34.620740 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d785995-0b74-48c5-9354-a524e7f382db None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:34.621033 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d785995-0b74-48c5-9354-a524e7f382db None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:34.623057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 154/307] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:34.624330 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d785995-0b74-48c5-9354-a524e7f382db None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:34.634467 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043fd805-58a5-4b60-bf94-6fe0156a76a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:34.635548 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043fd805-58a5-4b60-bf94-6fe0156a76a6 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:34.635835 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043fd805-58a5-4b60-bf94-6fe0156a76a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:34.636160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043fd805-58a5-4b60-bf94-6fe0156a76a6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:34.637367 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043fd805-58a5-4b60-bf94-6fe0156a76a6 None admin] RBAC: Adding query filter params (role.id=6b0968db45c64a66b001a39d40579011, user.id=161ef36cfa2f4420ad2be2a75ce51f01, scope.system=all) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:34.637691 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043fd805-58a5-4b60-bf94-6fe0156a76a6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:34.639615 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043fd805-58a5-4b60-bf94-6fe0156a76a6 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:34.649576 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 154/308] 10.222.0.42 () {56 vars in 1308 bytes} [Sun Sep 4 00:46:34 2022] GET /identity/v3/role_assignments?user.id=161ef36cfa2f4420ad2be2a75ce51f01&scope.system=all&role.id=6b0968db45c64a66b001a39d40579011 => generated 226 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:34.650641 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66009086-0e6b-48c5-b68b-cd7bd6546ce3 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:34.650929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66009086-0e6b-48c5-b68b-cd7bd6546ce3 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:34.651207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66009086-0e6b-48c5-b68b-cd7bd6546ce3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:34.674129 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-66009086-0e6b-48c5-b68b-cd7bd6546ce3 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:34.675378 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-66009086-0e6b-48c5-b68b-cd7bd6546ce3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:34.765304 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 155/309] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:34.773861 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d785995-0b74-48c5-9354-a524e7f382db None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:34.811580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a13675-8a89-4b63-b1c5-45a865459e39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:34.812198 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a13675-8a89-4b63-b1c5-45a865459e39 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:34.812479 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a13675-8a89-4b63-b1c5-45a865459e39 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:34.812751 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a13675-8a89-4b63-b1c5-45a865459e39 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:34.813456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a13675-8a89-4b63-b1c5-45a865459e39 None admin] RBAC: Adding query filter params (name=member) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:34.813765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a13675-8a89-4b63-b1c5-45a865459e39 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:34.815164 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a13675-8a89-4b63-b1c5-45a865459e39 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:34.824386 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 155/310] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:34 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:34.830295 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66009086-0e6b-48c5-b68b-cd7bd6546ce3 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:34.895603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:34.939438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66009086-0e6b-48c5-b68b-cd7bd6546ce3 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:34.976603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fd9db4-2e7c-42da-ab62-4e9a503d3e4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:34.977278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fd9db4-2e7c-42da-ab62-4e9a503d3e4f None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:34.977560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fd9db4-2e7c-42da-ab62-4e9a503d3e4f None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:34.977837 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fd9db4-2e7c-42da-ab62-4e9a503d3e4f None admin] PATH_INFO: `/v3/users/f54e8c9978684b00bc21d66dfa3397d9` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:34.978979 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fd9db4-2e7c-42da-ab62-4e9a503d3e4f None admin] RBAC: Authorizing `identity:get_user(user_id=f54e8c9978684b00bc21d66dfa3397d9)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:34.980431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fd9db4-2e7c-42da-ab62-4e9a503d3e4f None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:34.982037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 156/311] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:34 2022] GET /identity/v3/users/f54e8c9978684b00bc21d66dfa3397d9 => generated 293 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:34.987684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a13675-8a89-4b63-b1c5-45a865459e39 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:35.050185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:35.052021 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fd9db4-2e7c-42da-ab62-4e9a503d3e4f None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:35.105668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a13675-8a89-4b63-b1c5-45a865459e39 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:35.149148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e50b5c-0acf-43de-a33c-3b6fdc7c5ac9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:35.149872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e50b5c-0acf-43de-a33c-3b6fdc7c5ac9 None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:35.151134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e50b5c-0acf-43de-a33c-3b6fdc7c5ac9 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:35.151411 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e50b5c-0acf-43de-a33c-3b6fdc7c5ac9 None admin] PATH_INFO: `/v3/system/users/f54e8c9978684b00bc21d66dfa3397d9/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:35.153966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e50b5c-0acf-43de-a33c-3b6fdc7c5ac9 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=f54e8c9978684b00bc21d66dfa3397d9, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:35.156204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e50b5c-0acf-43de-a33c-3b6fdc7c5ac9 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:35.164491 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fd9db4-2e7c-42da-ab62-4e9a503d3e4f None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:35.182870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:35.200560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b75a07-121a-48e0-b16f-83ea22a1bdc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:35.201324 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b75a07-121a-48e0-b16f-83ea22a1bdc6 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:35.201606 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b75a07-121a-48e0-b16f-83ea22a1bdc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:35.201884 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b75a07-121a-48e0-b16f-83ea22a1bdc6 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:35.212903 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b75a07-121a-48e0-b16f-83ea22a1bdc6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:35.217935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-76b75a07-121a-48e0-b16f-83ea22a1bdc6 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:35.217935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:35.217935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:35.217935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:35.217935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:35.217935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:35.217935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:35.217935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:35.217935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:35.217935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:35.217935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:35.217935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:35.217935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:35.217935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:35.217935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:35.217935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:35.218779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:35.218779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:35.218779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:35.218779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:35.218779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:35.218779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:35.218779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:35.218779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:35.218779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:35.218779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:35.218779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:35.218779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:35.218779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:35.218779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:35.218779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:35.218779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:35.219554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:35.219554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:35.219554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:35.219554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:35.219554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:35.219554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:35.219554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:35.219554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:35.219554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:35.219554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:35.219554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:35.219554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:35.219554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 157/312] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:35 2022] GET /identity/v3/roles/reader => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:46:35.224891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b75a07-121a-48e0-b16f-83ea22a1bdc6 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:35.269105 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 156/313] 10.222.0.42 () {58 vars in 1334 bytes} [Sun Sep 4 00:46:34 2022] PUT /identity/v3/system/users/f54e8c9978684b00bc21d66dfa3397d9/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 00:46:35.349266 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b75a07-121a-48e0-b16f-83ea22a1bdc6 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:35.389946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc07a5f-2e99-4f07-9a02-16a7b62c40b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:35.390541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc07a5f-2e99-4f07-9a02-16a7b62c40b2 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:35.390830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc07a5f-2e99-4f07-9a02-16a7b62c40b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:35.391108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc07a5f-2e99-4f07-9a02-16a7b62c40b2 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:35.391788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc07a5f-2e99-4f07-9a02-16a7b62c40b2 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:35.392302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc07a5f-2e99-4f07-9a02-16a7b62c40b2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:35.393755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc07a5f-2e99-4f07-9a02-16a7b62c40b2 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:35.403592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 158/314] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:35 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:35.409198 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e50b5c-0acf-43de-a33c-3b6fdc7c5ac9 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:35.525505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e50b5c-0acf-43de-a33c-3b6fdc7c5ac9 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:35.546863 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:35.562162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8035c95e-1af7-49c1-88e9-322ae35d6fc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:35.563123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8035c95e-1af7-49c1-88e9-322ae35d6fc5 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:35.563600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8035c95e-1af7-49c1-88e9-322ae35d6fc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:35.564064 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8035c95e-1af7-49c1-88e9-322ae35d6fc5 None admin] PATH_INFO: `/v3/users/40cbd8a9da674f0e8b11d3946ca3a911` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:35.565580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8035c95e-1af7-49c1-88e9-322ae35d6fc5 None admin] RBAC: Authorizing `identity:get_user(user_id=40cbd8a9da674f0e8b11d3946ca3a911)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:35.567505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8035c95e-1af7-49c1-88e9-322ae35d6fc5 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:35.569516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 157/315] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:35 2022] GET /identity/v3/users/40cbd8a9da674f0e8b11d3946ca3a911 => generated 297 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:35.580256 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc07a5f-2e99-4f07-9a02-16a7b62c40b2 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:35.629937 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:35.696623 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc07a5f-2e99-4f07-9a02-16a7b62c40b2 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:35.736365 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3729229b-7b00-4bc9-9f5e-c1bdf92b8d71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:35.736986 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729229b-7b00-4bc9-9f5e-c1bdf92b8d71 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:35.737295 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729229b-7b00-4bc9-9f5e-c1bdf92b8d71 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:35.737574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729229b-7b00-4bc9-9f5e-c1bdf92b8d71 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:35.813032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3729229b-7b00-4bc9-9f5e-c1bdf92b8d71 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:35.814514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3729229b-7b00-4bc9-9f5e-c1bdf92b8d71 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:35.862095 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:35.923375 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 159/316] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:35.929734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8035c95e-1af7-49c1-88e9-322ae35d6fc5 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:36.036297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8035c95e-1af7-49c1-88e9-322ae35d6fc5 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:36.077919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02621930-c466-40f5-9739-fe6b243af467 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:36.078558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02621930-c466-40f5-9739-fe6b243af467 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:36.078839 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02621930-c466-40f5-9739-fe6b243af467 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:36.079116 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02621930-c466-40f5-9739-fe6b243af467 None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:36.080263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02621930-c466-40f5-9739-fe6b243af467 None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:36.081800 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02621930-c466-40f5-9739-fe6b243af467 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:36.084409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 158/317] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:35 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:36.090275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3729229b-7b00-4bc9-9f5e-c1bdf92b8d71 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:36.194000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3729229b-7b00-4bc9-9f5e-c1bdf92b8d71 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:36.229594 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b233eb5-4d9e-48d3-85ba-5a015c60c710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:36.230211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b233eb5-4d9e-48d3-85ba-5a015c60c710 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:36.230501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b233eb5-4d9e-48d3-85ba-5a015c60c710 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:36.230761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b233eb5-4d9e-48d3-85ba-5a015c60c710 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:36.231490 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b233eb5-4d9e-48d3-85ba-5a015c60c710 None admin] RBAC: Adding query filter params (role.id=6b0968db45c64a66b001a39d40579011, scope.project.id=d737f411ca514fb4aa93ee11d55926ae, user.id=40cbd8a9da674f0e8b11d3946ca3a911) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:36.231797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b233eb5-4d9e-48d3-85ba-5a015c60c710 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:36.233259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b233eb5-4d9e-48d3-85ba-5a015c60c710 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:36.239783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 160/318] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:46:36 2022] GET /identity/v3/role_assignments?user.id=40cbd8a9da674f0e8b11d3946ca3a911&scope.project.id=d737f411ca514fb4aa93ee11d55926ae&role.id=6b0968db45c64a66b001a39d40579011 => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:36.468506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ad519f-4d34-4288-842b-1bcc8817d5a6 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:36.468795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ad519f-4d34-4288-842b-1bcc8817d5a6 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:36.469095 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ad519f-4d34-4288-842b-1bcc8817d5a6 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:36.470692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 159/319] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:36.476677 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f31b8e0-50b7-4848-8f0f-5693a8e087df None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:36.476972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f31b8e0-50b7-4848-8f0f-5693a8e087df None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:36.477379 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f31b8e0-50b7-4848-8f0f-5693a8e087df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:36.484830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:36.495875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-3f31b8e0-50b7-4848-8f0f-5693a8e087df None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:36.497106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-3f31b8e0-50b7-4848-8f0f-5693a8e087df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:36.564572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 161/320] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:36.714637 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f70b5a-e64b-4368-b742-c4688c218cae None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:36.715158 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f70b5a-e64b-4368-b742-c4688c218cae None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:36.715158 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f70b5a-e64b-4368-b742-c4688c218cae None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:36.716676 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 160/321] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:36.721678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afe9b22-4ebf-4c1b-abfd-f9dd7c980ddb None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:36.722051 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afe9b22-4ebf-4c1b-abfd-f9dd7c980ddb None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:36.722215 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afe9b22-4ebf-4c1b-abfd-f9dd7c980ddb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:36.740805 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-7afe9b22-4ebf-4c1b-abfd-f9dd7c980ddb None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:36.742000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7afe9b22-4ebf-4c1b-abfd-f9dd7c980ddb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:36.810713 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 162/322] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:36.835823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f70b5a-e64b-4368-b742-c4688c218cae None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:36.895842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:36.944105 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f70b5a-e64b-4368-b742-c4688c218cae None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:36.983017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a639ef-c7cc-48b9-8e61-8463d5dcb7c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:36.983574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a639ef-c7cc-48b9-8e61-8463d5dcb7c9 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:36.983858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a639ef-c7cc-48b9-8e61-8463d5dcb7c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:36.984130 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a639ef-c7cc-48b9-8e61-8463d5dcb7c9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:36.985265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a639ef-c7cc-48b9-8e61-8463d5dcb7c9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:36.986704 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a639ef-c7cc-48b9-8e61-8463d5dcb7c9 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:36.989125 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 161/323] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:46:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:36.994074 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afe9b22-4ebf-4c1b-abfd-f9dd7c980ddb None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:37.058546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:37.088535 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a639ef-c7cc-48b9-8e61-8463d5dcb7c9 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:37.101350 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afe9b22-4ebf-4c1b-abfd-f9dd7c980ddb None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:37.136593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b74423-f0c4-4a13-b969-15dcd0b38003 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:37.137234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b74423-f0c4-4a13-b969-15dcd0b38003 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:37.137519 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b74423-f0c4-4a13-b969-15dcd0b38003 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:37.137791 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b74423-f0c4-4a13-b969-15dcd0b38003 None admin] PATH_INFO: `/v3/groups` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:37.138443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b74423-f0c4-4a13-b969-15dcd0b38003 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:37.139887 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b74423-f0c4-4a13-b969-15dcd0b38003 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:37.197187 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a639ef-c7cc-48b9-8e61-8463d5dcb7c9 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:37.220306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:37.230730 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd977f42-eb2f-4458-beec-2bbc4cbfa2bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:37.231357 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd977f42-eb2f-4458-beec-2bbc4cbfa2bc None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:37.231637 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd977f42-eb2f-4458-beec-2bbc4cbfa2bc None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:37.231905 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd977f42-eb2f-4458-beec-2bbc4cbfa2bc None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:37.245200 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd977f42-eb2f-4458-beec-2bbc4cbfa2bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:37.245466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 163/324] 10.222.0.42 () {60 vars in 1134 bytes} [Sun Sep 4 00:46:36 2022] POST /identity/v3/groups => generated 226 bytes in 251 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:46:37.252876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-bd977f42-eb2f-4458-beec-2bbc4cbfa2bc None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:37.252876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:37.252876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:37.252876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:37.252876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:37.252876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:37.252876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:37.252876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:37.252876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:37.252876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:37.252876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:37.252876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:37.252876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:37.252876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:37.252876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:37.252876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:37.254631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:37.254631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:37.254631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:37.254631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:37.254631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:37.254631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:37.254631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:37.254631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:37.254631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:37.254631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:37.254631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:37.254631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:37.254631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:37.254631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:37.254631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:37.254631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:37.255528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:37.255528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:37.255528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:37.255528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:37.255528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:37.255528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:37.255528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:37.255528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:37.255528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:37.255528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:37.255528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:37.255528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:37.258296 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 162/325] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:37 2022] GET /identity/v3/roles/reader => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:46:37.263788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b74423-f0c4-4a13-b969-15dcd0b38003 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:37.284806 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334dd547-000a-412c-a697-92753ea8d01a None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:37.285107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334dd547-000a-412c-a697-92753ea8d01a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:37.285426 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334dd547-000a-412c-a697-92753ea8d01a None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:37.286925 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 163/326] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:46:37.292574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f3f38e-3c42-4a1e-a6d4-4c7ec71c9769 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:37.292868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f3f38e-3c42-4a1e-a6d4-4c7ec71c9769 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:37.293580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f3f38e-3c42-4a1e-a6d4-4c7ec71c9769 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:37.312389 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-b3f3f38e-3c42-4a1e-a6d4-4c7ec71c9769 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:37.313694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b3f3f38e-3c42-4a1e-a6d4-4c7ec71c9769 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:37.383175 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b74423-f0c4-4a13-b969-15dcd0b38003 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:37.403038 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 164/327] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:46:37.409140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:37.420702 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d820cce-4786-435c-8f3b-6ef40325e96e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:37.421399 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d820cce-4786-435c-8f3b-6ef40325e96e None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:37.421673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d820cce-4786-435c-8f3b-6ef40325e96e None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:37.421945 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d820cce-4786-435c-8f3b-6ef40325e96e None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:37.422688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d820cce-4786-435c-8f3b-6ef40325e96e None admin] RBAC: Adding query filter params (name=reader) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:37.422968 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d820cce-4786-435c-8f3b-6ef40325e96e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:37.424362 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d820cce-4786-435c-8f3b-6ef40325e96e None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:37.434133 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 164/328] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:37 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:37.440291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f3f38e-3c42-4a1e-a6d4-4c7ec71c9769 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:37.559152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f3f38e-3c42-4a1e-a6d4-4c7ec71c9769 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:37.574325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:37.594372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3a4bd7-7ae7-49ba-a943-7cb149e1928b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:37.595009 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3a4bd7-7ae7-49ba-a943-7cb149e1928b None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:37.595352 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3a4bd7-7ae7-49ba-a943-7cb149e1928b None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:37.595632 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3a4bd7-7ae7-49ba-a943-7cb149e1928b None admin] PATH_INFO: `/v3/users/161ef36cfa2f4420ad2be2a75ce51f01` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:37.596789 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3a4bd7-7ae7-49ba-a943-7cb149e1928b None admin] RBAC: Authorizing `identity:get_user(user_id=161ef36cfa2f4420ad2be2a75ce51f01)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:37.598300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3a4bd7-7ae7-49ba-a943-7cb149e1928b None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:37.600150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 165/329] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:37 2022] GET /identity/v3/users/161ef36cfa2f4420ad2be2a75ce51f01 => generated 293 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:37.605863 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d820cce-4786-435c-8f3b-6ef40325e96e None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:37.679715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3a4bd7-7ae7-49ba-a943-7cb149e1928b None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:37.720172 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d820cce-4786-435c-8f3b-6ef40325e96e None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:37.760513 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281be078-034f-4e8c-8be4-95c58b8400d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:37.761280 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281be078-034f-4e8c-8be4-95c58b8400d0 None admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:37.761556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281be078-034f-4e8c-8be4-95c58b8400d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:37.761828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281be078-034f-4e8c-8be4-95c58b8400d0 None admin] PATH_INFO: `/v3/system/users/161ef36cfa2f4420ad2be2a75ce51f01/roles/6b0968db45c64a66b001a39d40579011` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:37.763233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281be078-034f-4e8c-8be4-95c58b8400d0 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=161ef36cfa2f4420ad2be2a75ce51f01, role_id=6b0968db45c64a66b001a39d40579011)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:37.764796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281be078-034f-4e8c-8be4-95c58b8400d0 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:37.800288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3a4bd7-7ae7-49ba-a943-7cb149e1928b None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:37.838156 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4897a6ec-c5af-4d9d-86fe-d38fe93c1891 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:37.838823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4897a6ec-c5af-4d9d-86fe-d38fe93c1891 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:37.839136 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4897a6ec-c5af-4d9d-86fe-d38fe93c1891 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:37.839403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4897a6ec-c5af-4d9d-86fe-d38fe93c1891 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:37.841723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 165/330] 10.222.0.42 () {58 vars in 1334 bytes} [Sun Sep 4 00:46:37 2022] PUT /identity/v3/system/users/161ef36cfa2f4420ad2be2a75ce51f01/roles/6b0968db45c64a66b001a39d40579011 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:46:37.857063 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4897a6ec-c5af-4d9d-86fe-d38fe93c1891 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:37.862633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-4897a6ec-c5af-4d9d-86fe-d38fe93c1891 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:37.862633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:37.862633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:37.862633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:37.862633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:37.862633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:37.862633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:37.862633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:37.862633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:37.862633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:37.862633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:37.862633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:37.862633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:37.862633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:37.862633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:37.862633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:37.863620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:37.863620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:37.863620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:37.863620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:37.863620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:37.863620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:37.863620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:37.863620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:37.863620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:37.863620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:37.863620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:37.863620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:37.863620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:37.863620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:37.863620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:37.863620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:37.864903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:37.864903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:37.864903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:37.864903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:37.864903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:37.864903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:37.864903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:37.864903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:37.864903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:37.864903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:37.864903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:37.864903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:37.868740 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 166/331] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:46:37.870215 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281be078-034f-4e8c-8be4-95c58b8400d0 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:37.934250 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:37.994267 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281be078-034f-4e8c-8be4-95c58b8400d0 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:38.031857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e706e12-81da-449e-abb3-a94fbd91a4fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:38.032460 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e706e12-81da-449e-abb3-a94fbd91a4fe None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:38.032737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e706e12-81da-449e-abb3-a94fbd91a4fe None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:38.033007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e706e12-81da-449e-abb3-a94fbd91a4fe None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:38.033699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e706e12-81da-449e-abb3-a94fbd91a4fe None admin] RBAC: Adding query filter params (name=member) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:38.033997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e706e12-81da-449e-abb3-a94fbd91a4fe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:38.035972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e706e12-81da-449e-abb3-a94fbd91a4fe None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:38.044760 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 166/332] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:37 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:38.050440 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4897a6ec-c5af-4d9d-86fe-d38fe93c1891 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:38.097613 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:38.157434 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4897a6ec-c5af-4d9d-86fe-d38fe93c1891 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:38.195739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafb700c-0bfb-4d30-bae8-e8a1d12b0784 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:38.196436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafb700c-0bfb-4d30-bae8-e8a1d12b0784 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:38.196718 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafb700c-0bfb-4d30-bae8-e8a1d12b0784 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:38.196987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafb700c-0bfb-4d30-bae8-e8a1d12b0784 None admin] PATH_INFO: `/v3/users/f54e8c9978684b00bc21d66dfa3397d9` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:38.198257 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dafb700c-0bfb-4d30-bae8-e8a1d12b0784 None admin] RBAC: Authorizing `identity:get_user(user_id=f54e8c9978684b00bc21d66dfa3397d9)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:38.199748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dafb700c-0bfb-4d30-bae8-e8a1d12b0784 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:38.201346 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 167/333] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:38 2022] GET /identity/v3/users/f54e8c9978684b00bc21d66dfa3397d9 => generated 293 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:38.208258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e706e12-81da-449e-abb3-a94fbd91a4fe None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:38.322595 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e706e12-81da-449e-abb3-a94fbd91a4fe None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:38.339757 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:38.366164 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13adcd81-e0c4-427b-9f2d-b2a942090f26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:38.366804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13adcd81-e0c4-427b-9f2d-b2a942090f26 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:38.367185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13adcd81-e0c4-427b-9f2d-b2a942090f26 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:38.367525 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13adcd81-e0c4-427b-9f2d-b2a942090f26 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:38.368385 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13adcd81-e0c4-427b-9f2d-b2a942090f26 None admin] RBAC: Adding query filter params (role.id=4bfd6d0544ee4a9e933538e78135a12c, user.id=f54e8c9978684b00bc21d66dfa3397d9, scope.system=all) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:38.368809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13adcd81-e0c4-427b-9f2d-b2a942090f26 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:38.370358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13adcd81-e0c4-427b-9f2d-b2a942090f26 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:38.388539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 167/334] 10.222.0.42 () {56 vars in 1308 bytes} [Sun Sep 4 00:46:38 2022] GET /identity/v3/role_assignments?user.id=f54e8c9978684b00bc21d66dfa3397d9&scope.system=all&role.id=4bfd6d0544ee4a9e933538e78135a12c => generated 510 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:39.643887 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01de8a9b-0b31-497f-9dfb-ba3f500c4692 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:39.644400 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01de8a9b-0b31-497f-9dfb-ba3f500c4692 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:39.644476 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01de8a9b-0b31-497f-9dfb-ba3f500c4692 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:39.646849 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 168/335] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:39.652540 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cdf81f-6bb8-40b3-a4e3-a91bbf1a4911 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:39.653118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cdf81f-6bb8-40b3-a4e3-a91bbf1a4911 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:39.653118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cdf81f-6bb8-40b3-a4e3-a91bbf1a4911 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:39.679439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-82cdf81f-6bb8-40b3-a4e3-a91bbf1a4911 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:39.680671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-82cdf81f-6bb8-40b3-a4e3-a91bbf1a4911 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:39.780984 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 168/336] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:39.834602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966f14f2-bbb0-45a1-9828-e51af517565a None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:39.835404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966f14f2-bbb0-45a1-9828-e51af517565a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:39.835895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966f14f2-bbb0-45a1-9828-e51af517565a None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:39.838155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 169/337] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:39 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:39.843482 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87eed3e6-4cab-443f-a922-da662f58d924 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:39.843769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87eed3e6-4cab-443f-a922-da662f58d924 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:39.844050 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87eed3e6-4cab-443f-a922-da662f58d924 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:39.867515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-87eed3e6-4cab-443f-a922-da662f58d924 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:39.868701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-87eed3e6-4cab-443f-a922-da662f58d924 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:39.949397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 169/338] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:40.320381 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105d4c7a-79b1-455d-9f2b-1b0bf9529a9d None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:40.320729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105d4c7a-79b1-455d-9f2b-1b0bf9529a9d None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:40.321014 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105d4c7a-79b1-455d-9f2b-1b0bf9529a9d None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:40.322640 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 170/339] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:40.326115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87eed3e6-4cab-443f-a922-da662f58d924 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:40.342688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10719eeb-e506-4fe2-acec-b20a13e31d14 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:40.350456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10719eeb-e506-4fe2-acec-b20a13e31d14 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:40.350456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10719eeb-e506-4fe2-acec-b20a13e31d14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:40.364478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-10719eeb-e506-4fe2-acec-b20a13e31d14 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:40.365707 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-10719eeb-e506-4fe2-acec-b20a13e31d14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:40.381503 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:40.473516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 171/340] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Sep 04 00:46:40.499839 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87eed3e6-4cab-443f-a922-da662f58d924 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:40.536407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10719eeb-e506-4fe2-acec-b20a13e31d14 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:40.542814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a96bab5-778a-46bf-96a7-59c9da2b02ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:40.543671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a96bab5-778a-46bf-96a7-59c9da2b02ca None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:40.544089 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a96bab5-778a-46bf-96a7-59c9da2b02ca None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:40.544638 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a96bab5-778a-46bf-96a7-59c9da2b02ca None admin] PATH_INFO: `/v3/roles/member` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:40.557442 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a96bab5-778a-46bf-96a7-59c9da2b02ca None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:40.563203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-4a96bab5-778a-46bf-96a7-59c9da2b02ca None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:40.563203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:40.563203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:40.563203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:40.563203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:40.563203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:40.563203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:40.563203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:40.563203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:40.563203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:40.563203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:40.563203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:40.563203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:40.563203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:40.563203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:40.563203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:40.564233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:40.564233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:40.564233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:40.564233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:40.564233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:40.564233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:40.564233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:40.564233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:40.564233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:40.564233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:40.564233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:40.564233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:40.564233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:40.564233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:40.564233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:40.564233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:40.565218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:40.565218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:40.565218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:40.565218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:40.565218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:40.565218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:40.565218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:40.565218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:40.565218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:40.565218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:40.565218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:40.565218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:40.568280 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 170/341] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:40 2022] GET /identity/v3/roles/member => generated 84 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:40.575534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a96bab5-778a-46bf-96a7-59c9da2b02ca None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:40.640597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:40.686173 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:40.697982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a96bab5-778a-46bf-96a7-59c9da2b02ca None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:40.736396 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082c64f3-fb40-4e2a-985d-57672178ec56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:40.737340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082c64f3-fb40-4e2a-985d-57672178ec56 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:40.737793 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082c64f3-fb40-4e2a-985d-57672178ec56 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:40.738756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082c64f3-fb40-4e2a-985d-57672178ec56 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:40.739702 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082c64f3-fb40-4e2a-985d-57672178ec56 None admin] RBAC: Adding query filter params (name=member) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:40.740220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082c64f3-fb40-4e2a-985d-57672178ec56 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:40.741798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082c64f3-fb40-4e2a-985d-57672178ec56 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:40.748580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10719eeb-e506-4fe2-acec-b20a13e31d14 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:40.753817 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 171/342] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:40 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:40.783806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082c64f3-fb40-4e2a-985d-57672178ec56 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:40.797142 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a988f095-4429-4906-9622-d40601e8eac6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:40.797142 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a988f095-4429-4906-9622-d40601e8eac6 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:40.797142 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a988f095-4429-4906-9622-d40601e8eac6 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:40.797142 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a988f095-4429-4906-9622-d40601e8eac6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:40.809126 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a988f095-4429-4906-9622-d40601e8eac6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:40.812329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-a988f095-4429-4906-9622-d40601e8eac6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:40.812329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:40.812329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:40.812329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:40.812329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:40.812329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:40.812329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:40.812329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:40.812329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:40.812329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:40.812329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:40.812329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:40.812329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:40.812329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:40.812329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:40.812329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:40.813249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:40.813249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:40.813249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:40.813249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:40.813249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:40.813249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:40.813249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:40.813249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:40.813249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:40.813249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:40.813249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:40.813249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:40.813249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:40.813249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:40.813249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:40.813249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:40.814250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:40.814250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:40.814250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:40.814250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:40.814250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:40.814250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:40.814250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:40.814250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:40.814250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:40.814250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:40.814250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:40.814250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:40.816543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 172/343] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:46:40 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:46:40.822842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a988f095-4429-4906-9622-d40601e8eac6 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:40.925260 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082c64f3-fb40-4e2a-985d-57672178ec56 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:40.961204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bfd5d3-fe9d-4105-9afe-949f59b168b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:40.961833 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bfd5d3-fe9d-4105-9afe-949f59b168b8 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:40.962178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bfd5d3-fe9d-4105-9afe-949f59b168b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:40.962599 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bfd5d3-fe9d-4105-9afe-949f59b168b8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:40.972204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:40.976739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a988f095-4429-4906-9622-d40601e8eac6 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:40.991275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:41.030878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7e8117-9739-46a6-8169-34c586c13cf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:41.031597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7e8117-9739-46a6-8169-34c586c13cf6 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:41.031912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7e8117-9739-46a6-8169-34c586c13cf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:41.032190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7e8117-9739-46a6-8169-34c586c13cf6 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:41.032874 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7e8117-9739-46a6-8169-34c586c13cf6 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:41.033208 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7e8117-9739-46a6-8169-34c586c13cf6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:41.034597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7e8117-9739-46a6-8169-34c586c13cf6 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:41.045033 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4bfd5d3-fe9d-4105-9afe-949f59b168b8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:41.046897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4bfd5d3-fe9d-4105-9afe-949f59b168b8 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:41.048451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 173/344] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:46:40 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:41.054433 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7e8117-9739-46a6-8169-34c586c13cf6 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:41.164568 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 172/345] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 04 00:46:41.166183 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a175e9c4-f0d8-4bb1-8693-e16e34871396 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:41.166660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a175e9c4-f0d8-4bb1-8693-e16e34871396 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:41.167352 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a175e9c4-f0d8-4bb1-8693-e16e34871396 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:41.168877 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 173/346] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:41.173276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a175e9c4-f0d8-4bb1-8693-e16e34871396 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:41.194935 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7e8117-9739-46a6-8169-34c586c13cf6 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:41.248957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c257b76e-add2-483c-9623-3b52976cd95e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:41.249648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c257b76e-add2-483c-9623-3b52976cd95e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:41.249931 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c257b76e-add2-483c-9623-3b52976cd95e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:41.250214 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c257b76e-add2-483c-9623-3b52976cd95e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:41.255423 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:41.284003 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:41.319136 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a175e9c4-f0d8-4bb1-8693-e16e34871396 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:41.334311 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c257b76e-add2-483c-9623-3b52976cd95e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:41.336228 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c257b76e-add2-483c-9623-3b52976cd95e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:41.356600 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6c672a-dfcc-4c78-914e-abde71d764d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:41.357395 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6c672a-dfcc-4c78-914e-abde71d764d2 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:41.357900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6c672a-dfcc-4c78-914e-abde71d764d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:41.358045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6c672a-dfcc-4c78-914e-abde71d764d2 None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:41.359260 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6c672a-dfcc-4c78-914e-abde71d764d2 None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:41.360946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6c672a-dfcc-4c78-914e-abde71d764d2 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:41.362715 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 174/347] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:41 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:41.364935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b01aaf-6774-4d8d-bfa4-4802fad45e86 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:41.369367 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b01aaf-6774-4d8d-bfa4-4802fad45e86 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:41.370043 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b01aaf-6774-4d8d-bfa4-4802fad45e86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:41.393571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-10b01aaf-6774-4d8d-bfa4-4802fad45e86 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:41.394998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-10b01aaf-6774-4d8d-bfa4-4802fad45e86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:41.455668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 174/348] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 04 00:46:41.457597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c257b76e-add2-483c-9623-3b52976cd95e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:41.470179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 175/349] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:41.471130 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b01aaf-6774-4d8d-bfa4-4802fad45e86 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:41.526532 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:41.528621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:41.571666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c257b76e-add2-483c-9623-3b52976cd95e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:41.612780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34f5c4c-91b8-4d81-be88-70016c2fd20b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:41.613331 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b01aaf-6774-4d8d-bfa4-4802fad45e86 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:41.614029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34f5c4c-91b8-4d81-be88-70016c2fd20b None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:41.614484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34f5c4c-91b8-4d81-be88-70016c2fd20b None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:41.614954 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34f5c4c-91b8-4d81-be88-70016c2fd20b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:41.626529 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34f5c4c-91b8-4d81-be88-70016c2fd20b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:41.633700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-d34f5c4c-91b8-4d81-be88-70016c2fd20b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:41.633700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:41.633700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:41.633700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:41.633700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:41.633700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:41.633700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:41.633700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:41.633700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:41.633700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:41.633700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:41.633700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:41.633700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:41.633700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:41.633700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:41.633700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:41.634607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:41.634607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:41.634607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:41.634607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:41.634607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:41.634607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:41.634607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:41.634607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:41.634607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:41.634607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:41.634607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:41.634607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:41.634607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:41.634607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:41.634607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:41.634607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:41.635409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:41.635409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:41.635409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:41.635409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:41.635409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:41.635409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:41.635409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:41.635409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:41.635409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:41.635409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:41.635409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 04 00:46:41.635409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:41.637528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 175/350] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:41 2022] GET /identity/v3/roles/reader => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:41.638898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34f5c4c-91b8-4d81-be88-70016c2fd20b None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:41.678376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dadaf4-fd36-4c5e-bd0b-6a4506bd2b0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:41.679168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dadaf4-fd36-4c5e-bd0b-6a4506bd2b0c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:41.679549 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dadaf4-fd36-4c5e-bd0b-6a4506bd2b0c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:41.679927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dadaf4-fd36-4c5e-bd0b-6a4506bd2b0c None admin] PATH_INFO: `/v3/groups/bba303b549124f5898fa58d1e1f31db9` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:41.692190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dadaf4-fd36-4c5e-bd0b-6a4506bd2b0c None admin] RBAC: Authorizing `identity:get_group(group_id=bba303b549124f5898fa58d1e1f31db9)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:41.694182 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dadaf4-fd36-4c5e-bd0b-6a4506bd2b0c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:41.695982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 176/351] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 4 00:46:41 2022] GET /identity/v3/groups/bba303b549124f5898fa58d1e1f31db9 => generated 226 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:41.696842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dadaf4-fd36-4c5e-bd0b-6a4506bd2b0c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:41.755520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34f5c4c-91b8-4d81-be88-70016c2fd20b None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:41.781451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:41.799855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e526167-26e2-4c00-b778-97675c81969a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:41.800651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e526167-26e2-4c00-b778-97675c81969a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:41.801113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e526167-26e2-4c00-b778-97675c81969a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:41.801750 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e526167-26e2-4c00-b778-97675c81969a None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:41.803078 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e526167-26e2-4c00-b778-97675c81969a None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:41.804707 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e526167-26e2-4c00-b778-97675c81969a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:41.806851 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 176/352] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:41 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:41.807733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e526167-26e2-4c00-b778-97675c81969a None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:41.861230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dadaf4-fd36-4c5e-bd0b-6a4506bd2b0c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:41.877666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:41.913658 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac58767-139e-420d-a046-1fd37a465fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:41.914372 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac58767-139e-420d-a046-1fd37a465fb9 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:41.914727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac58767-139e-420d-a046-1fd37a465fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:41.915256 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac58767-139e-420d-a046-1fd37a465fb9 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:41.916434 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac58767-139e-420d-a046-1fd37a465fb9 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:41.916762 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac58767-139e-420d-a046-1fd37a465fb9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:41.918524 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac58767-139e-420d-a046-1fd37a465fb9 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:41.930235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 177/353] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:41 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:41.931515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a1f116-249f-447e-92e4-4786e6a2f35f None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:41.931808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a1f116-249f-447e-92e4-4786e6a2f35f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:41.932084 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a1f116-249f-447e-92e4-4786e6a2f35f None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:41.937672 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 178/354] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:41 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:41.938466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a1f116-249f-447e-92e4-4786e6a2f35f None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:41.941454 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e526167-26e2-4c00-b778-97675c81969a None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:41.985467 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adf0099-3eca-4de6-95e2-5e6aba00281c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:41.986090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adf0099-3eca-4de6-95e2-5e6aba00281c None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:41.986379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adf0099-3eca-4de6-95e2-5e6aba00281c None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:41.986653 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adf0099-3eca-4de6-95e2-5e6aba00281c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:41.987398 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9adf0099-3eca-4de6-95e2-5e6aba00281c None admin] RBAC: Adding query filter params (group.id=bba303b549124f5898fa58d1e1f31db9, role.id=4bfd6d0544ee4a9e933538e78135a12c, scope.project.id=1193eb16a76d432faf6141e47cc99c5a) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:41.987711 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9adf0099-3eca-4de6-95e2-5e6aba00281c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:41.989629 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9adf0099-3eca-4de6-95e2-5e6aba00281c None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:42.005128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 177/355] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 4 00:46:41 2022] GET /identity/v3/role_assignments?group.id=bba303b549124f5898fa58d1e1f31db9&scope.project.id=1193eb16a76d432faf6141e47cc99c5a&role.id=4bfd6d0544ee4a9e933538e78135a12c => generated 260 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:42.005128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adf0099-3eca-4de6-95e2-5e6aba00281c None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:42.077471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:42.078821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a1f116-249f-447e-92e4-4786e6a2f35f None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:42.136845 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d033f9-fbc9-4973-88d5-965a9a0406de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:42.137536 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d033f9-fbc9-4973-88d5-965a9a0406de None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:42.137820 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d033f9-fbc9-4973-88d5-965a9a0406de None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:42.138093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d033f9-fbc9-4973-88d5-965a9a0406de None admin] PATH_INFO: `/v3/groups/bba303b549124f5898fa58d1e1f31db9` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:42.140023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d033f9-fbc9-4973-88d5-965a9a0406de None admin] RBAC: Authorizing `identity:get_group(group_id=bba303b549124f5898fa58d1e1f31db9)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:42.141830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d033f9-fbc9-4973-88d5-965a9a0406de None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:42.142959 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adf0099-3eca-4de6-95e2-5e6aba00281c None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:42.143810 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 179/356] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 4 00:46:41 2022] GET /identity/v3/groups/bba303b549124f5898fa58d1e1f31db9 => generated 226 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:42.149112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d033f9-fbc9-4973-88d5-965a9a0406de None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:42.205374 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d304d69e-1f03-40d1-b383-f6e00f35d3ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:42.206068 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d304d69e-1f03-40d1-b383-f6e00f35d3ea None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:42.206350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d304d69e-1f03-40d1-b383-f6e00f35d3ea None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:42.206681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d304d69e-1f03-40d1-b383-f6e00f35d3ea None admin] PATH_INFO: `/v3/roles/member` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:42.216037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:42.222658 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d304d69e-1f03-40d1-b383-f6e00f35d3ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:42.233373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-d304d69e-1f03-40d1-b383-f6e00f35d3ea None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:42.233373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:42.233373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:42.233373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:42.233373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:42.233373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:42.233373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:42.233373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:42.233373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:42.233373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:42.233373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:42.233373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:42.233373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:42.233373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:42.233373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:42.233373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:42.234276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:42.234276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:42.234276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:42.234276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:42.234276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:42.234276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:42.234276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:42.234276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:42.234276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:42.234276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:42.234276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:42.234276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:42.234276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:42.234276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:42.234276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:42.234276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:42.235037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:42.235037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:42.235037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:42.235037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:42.235037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:42.235037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:42.235037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:42.235037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:42.235037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:42.235037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:42.235037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:42.235037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:42.235037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 178/357] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:42.236271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc115415-2aea-4ea0-86e9-775d95c24900 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:42.240619 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc115415-2aea-4ea0-86e9-775d95c24900 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:42.240619 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc115415-2aea-4ea0-86e9-775d95c24900 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:42.265363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-cc115415-2aea-4ea0-86e9-775d95c24900 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:42.267371 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-cc115415-2aea-4ea0-86e9-775d95c24900 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:42.288251 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d033f9-fbc9-4973-88d5-965a9a0406de None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:42.335620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6787cc5d-9025-4640-9b32-197583bf816d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:42.336242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6787cc5d-9025-4640-9b32-197583bf816d None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:42.336524 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6787cc5d-9025-4640-9b32-197583bf816d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:42.336823 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6787cc5d-9025-4640-9b32-197583bf816d None admin] PATH_INFO: `/v3/users/161ef36cfa2f4420ad2be2a75ce51f01` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:42.338209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6787cc5d-9025-4640-9b32-197583bf816d None admin] RBAC: Authorizing `identity:get_user(user_id=161ef36cfa2f4420ad2be2a75ce51f01)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:42.339750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6787cc5d-9025-4640-9b32-197583bf816d None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:42.342473 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 180/358] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:46:42 2022] GET /identity/v3/users/161ef36cfa2f4420ad2be2a75ce51f01 => generated 293 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:42.343954 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6787cc5d-9025-4640-9b32-197583bf816d None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:42.361789 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 179/359] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:42.363998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc115415-2aea-4ea0-86e9-775d95c24900 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:42.420399 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:42.502098 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc115415-2aea-4ea0-86e9-775d95c24900 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:42.510257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6787cc5d-9025-4640-9b32-197583bf816d None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:42.525365 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:42.546952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1400bf-fa74-43dc-89ae-56a37bd561ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:42.547807 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1400bf-fa74-43dc-89ae-56a37bd561ea None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:42.548218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1400bf-fa74-43dc-89ae-56a37bd561ea None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:42.548649 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1400bf-fa74-43dc-89ae-56a37bd561ea None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:42.549504 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1400bf-fa74-43dc-89ae-56a37bd561ea None admin] RBAC: Adding query filter params (name=member) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:42.549968 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1400bf-fa74-43dc-89ae-56a37bd561ea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:42.551894 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dc80d5-a616-4cbf-8ea8-a842b14d7677 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:42.552227 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1400bf-fa74-43dc-89ae-56a37bd561ea None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:42.552763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dc80d5-a616-4cbf-8ea8-a842b14d7677 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:42.553740 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dc80d5-a616-4cbf-8ea8-a842b14d7677 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:42.553740 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dc80d5-a616-4cbf-8ea8-a842b14d7677 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:42.554343 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dc80d5-a616-4cbf-8ea8-a842b14d7677 None admin] RBAC: Adding query filter params (group.id=bba303b549124f5898fa58d1e1f31db9, role.id=c599b217579e42a49c657c642ea0648f, scope.project.id=1193eb16a76d432faf6141e47cc99c5a) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:42.554752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dc80d5-a616-4cbf-8ea8-a842b14d7677 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:42.556274 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dc80d5-a616-4cbf-8ea8-a842b14d7677 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:42.563669 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 181/361] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 4 00:46:42 2022] GET /identity/v3/role_assignments?group.id=bba303b549124f5898fa58d1e1f31db9&scope.project.id=1193eb16a76d432faf6141e47cc99c5a&role.id=c599b217579e42a49c657c642ea0648f => generated 260 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:42.563896 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 180/361] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:42.564769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1400bf-fa74-43dc-89ae-56a37bd561ea None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:42.565578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a1801a-f0b2-4c70-a72e-3b6b92d7585b None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:42.566108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a1801a-f0b2-4c70-a72e-3b6b92d7585b None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:42.567093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a1801a-f0b2-4c70-a72e-3b6b92d7585b None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:42.573011 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 182/362] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:42.574225 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a1801a-f0b2-4c70-a72e-3b6b92d7585b None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:42.688857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a1801a-f0b2-4c70-a72e-3b6b92d7585b None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:42.752922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3cd2d9-813f-4600-a636-3424dc1b7359 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:42.752922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3cd2d9-813f-4600-a636-3424dc1b7359 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:42.752922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3cd2d9-813f-4600-a636-3424dc1b7359 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:42.752922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3cd2d9-813f-4600-a636-3424dc1b7359 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:42.774722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1400bf-fa74-43dc-89ae-56a37bd561ea None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:42.783236 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:42.814722 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:42.828138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2be29e-e726-40fc-a92e-3588ad1b203a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:42.829228 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2be29e-e726-40fc-a92e-3588ad1b203a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:42.829788 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2be29e-e726-40fc-a92e-3588ad1b203a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:42.830260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2be29e-e726-40fc-a92e-3588ad1b203a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:42.831190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2be29e-e726-40fc-a92e-3588ad1b203a None admin] RBAC: Adding query filter params (role.id=6b0968db45c64a66b001a39d40579011, user.id=161ef36cfa2f4420ad2be2a75ce51f01, scope.system=all) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:42.831715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2be29e-e726-40fc-a92e-3588ad1b203a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:42.834152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2be29e-e726-40fc-a92e-3588ad1b203a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:42.848712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 181/363] 10.222.0.42 () {56 vars in 1308 bytes} [Sun Sep 4 00:46:42 2022] GET /identity/v3/role_assignments?user.id=161ef36cfa2f4420ad2be2a75ce51f01&scope.system=all&role.id=6b0968db45c64a66b001a39d40579011 => generated 510 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:42.850026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3cd2d9-813f-4600-a636-3424dc1b7359 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:42.850574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957141e9-0e8e-41ff-a65e-ea9a77415c9a None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:42.851069 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957141e9-0e8e-41ff-a65e-ea9a77415c9a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:42.851502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3cd2d9-813f-4600-a636-3424dc1b7359 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:42.851764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957141e9-0e8e-41ff-a65e-ea9a77415c9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:42.879493 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-957141e9-0e8e-41ff-a65e-ea9a77415c9a None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:42.881230 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-957141e9-0e8e-41ff-a65e-ea9a77415c9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:42.970680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 182/364] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:42.971866 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957141e9-0e8e-41ff-a65e-ea9a77415c9a None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:43.010074 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 183/365] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:43.016837 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3cd2d9-813f-4600-a636-3424dc1b7359 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:43.149543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:43.161868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957141e9-0e8e-41ff-a65e-ea9a77415c9a None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:43.189913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:43.221530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9001b4b-f111-4094-a66f-2a78dc130679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:43.223336 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9001b4b-f111-4094-a66f-2a78dc130679 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:43.224171 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9001b4b-f111-4094-a66f-2a78dc130679 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:43.224722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9001b4b-f111-4094-a66f-2a78dc130679 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:43.239881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9001b4b-f111-4094-a66f-2a78dc130679 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:43.240681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3cd2d9-813f-4600-a636-3424dc1b7359 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:43.245520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-b9001b4b-f111-4094-a66f-2a78dc130679 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:43.245520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:43.245520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:43.245520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:43.245520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:43.245520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:43.245520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:43.245520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:43.245520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:43.245520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:43.245520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:43.245520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:43.245520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:43.245520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:43.245520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:43.245520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:43.246443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:43.246443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:43.246443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:43.246443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:43.246443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:43.246443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:43.246443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:43.246443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:43.246443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:43.246443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:43.246443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:43.246443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:43.246443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:43.246443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:43.246443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:43.246443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:43.247231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:43.247231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:43.247231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:43.247231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:43.247231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:43.247231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:43.247231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:43.247231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:43.247231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:43.247231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:43.247231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:43.247231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:43.250729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 183/366] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:46:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:43.257634 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9001b4b-f111-4094-a66f-2a78dc130679 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:43.345786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c6fb98-250c-4d88-a3ef-d43d0f26cce3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:43.346435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c6fb98-250c-4d88-a3ef-d43d0f26cce3 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:43.346710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c6fb98-250c-4d88-a3ef-d43d0f26cce3 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:43.346981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c6fb98-250c-4d88-a3ef-d43d0f26cce3 None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:43.348417 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c6fb98-250c-4d88-a3ef-d43d0f26cce3 None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:43.350070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23c6fb98-250c-4d88-a3ef-d43d0f26cce3 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:43.351726 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 184/367] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:43 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 335 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:43.374626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c6fb98-250c-4d88-a3ef-d43d0f26cce3 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:43.381176 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9001b4b-f111-4094-a66f-2a78dc130679 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:43.418845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a30706-a064-4d17-b24c-199310144d5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:43.419591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a30706-a064-4d17-b24c-199310144d5a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:43.419872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a30706-a064-4d17-b24c-199310144d5a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:43.420144 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a30706-a064-4d17-b24c-199310144d5a None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:43.420843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a30706-a064-4d17-b24c-199310144d5a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:43.421169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a30706-a064-4d17-b24c-199310144d5a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:43.422598 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a30706-a064-4d17-b24c-199310144d5a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:43.432572 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 184/368] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:46:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:43.439088 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a30706-a064-4d17-b24c-199310144d5a None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:43.463554 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:43.542043 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c6fb98-250c-4d88-a3ef-d43d0f26cce3 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:43.563809 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a30706-a064-4d17-b24c-199310144d5a None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:43.579343 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1bad33-7d53-4585-9f62-a48cb5a3fd43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:43.580097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1bad33-7d53-4585-9f62-a48cb5a3fd43 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:43.580376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1bad33-7d53-4585-9f62-a48cb5a3fd43 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:43.580648 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1bad33-7d53-4585-9f62-a48cb5a3fd43 None admin] PATH_INFO: `/v3/groups/bba303b549124f5898fa58d1e1f31db9` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:43.581874 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1bad33-7d53-4585-9f62-a48cb5a3fd43 None admin] RBAC: Authorizing `identity:get_group(group_id=bba303b549124f5898fa58d1e1f31db9)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:43.583899 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1bad33-7d53-4585-9f62-a48cb5a3fd43 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:43.585597 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 185/369] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 4 00:46:43 2022] GET /identity/v3/groups/bba303b549124f5898fa58d1e1f31db9 => generated 226 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:43.589967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1bad33-7d53-4585-9f62-a48cb5a3fd43 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:43.605881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75a25da-1f0f-44ac-980f-7f90b8498581 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:43.606819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75a25da-1f0f-44ac-980f-7f90b8498581 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:43.607301 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75a25da-1f0f-44ac-980f-7f90b8498581 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:43.607795 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:43.608301 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75a25da-1f0f-44ac-980f-7f90b8498581 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:43.686705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75a25da-1f0f-44ac-980f-7f90b8498581 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:43.688689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75a25da-1f0f-44ac-980f-7f90b8498581 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:43.700287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1bad33-7d53-4585-9f62-a48cb5a3fd43 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:43.709695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:43.737454 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5894206-2e45-4966-9c84-d30470c8549b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:43.738876 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5894206-2e45-4966-9c84-d30470c8549b None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:43.739294 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5894206-2e45-4966-9c84-d30470c8549b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:43.739698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5894206-2e45-4966-9c84-d30470c8549b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:43.751989 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5894206-2e45-4966-9c84-d30470c8549b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:43.757759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-d5894206-2e45-4966-9c84-d30470c8549b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:43.757759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:43.757759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:43.757759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:43.757759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:43.757759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:43.757759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:43.757759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:43.757759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:43.757759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:43.757759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:43.757759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:43.757759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:43.757759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:43.757759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:43.757759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:43.759016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:43.759016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:43.759016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:43.759016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:43.759016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:43.759016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:43.759016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:43.759016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:43.759016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:43.759016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:43.759016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:43.759016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:43.759016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:43.759016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:43.759016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:43.759016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:43.759798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:43.759798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:43.759798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:43.759798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:43.759798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:43.759798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:43.759798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:43.759798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:43.759798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:43.759798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:43.759798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:43.759798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:43.767291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 186/370] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:46:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:43.768541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5894206-2e45-4966-9c84-d30470c8549b None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:43.808275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 185/371] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 04 00:46:43.809253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75a25da-1f0f-44ac-980f-7f90b8498581 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:43.847444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:43.880798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5894206-2e45-4966-9c84-d30470c8549b None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:43.918378 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9d1b1e-a672-4520-a0a6-5d42d5abb14f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:43.920087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9d1b1e-a672-4520-a0a6-5d42d5abb14f None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:43.920365 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9d1b1e-a672-4520-a0a6-5d42d5abb14f None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:43.920633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9d1b1e-a672-4520-a0a6-5d42d5abb14f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:43.921391 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9d1b1e-a672-4520-a0a6-5d42d5abb14f None admin] RBAC: Adding query filter params (group.id=bba303b549124f5898fa58d1e1f31db9, role.id=4bfd6d0544ee4a9e933538e78135a12c, scope.project.id=d737f411ca514fb4aa93ee11d55926ae) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:43.921715 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9d1b1e-a672-4520-a0a6-5d42d5abb14f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:43.923116 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9d1b1e-a672-4520-a0a6-5d42d5abb14f None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:43.933212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 187/372] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 4 00:46:43 2022] GET /identity/v3/role_assignments?group.id=bba303b549124f5898fa58d1e1f31db9&scope.project.id=d737f411ca514fb4aa93ee11d55926ae&role.id=4bfd6d0544ee4a9e933538e78135a12c => generated 260 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:43.934183 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9d1b1e-a672-4520-a0a6-5d42d5abb14f None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:43.934953 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75a25da-1f0f-44ac-980f-7f90b8498581 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:43.965502 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:43.983831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e1a0b5-ffea-4df8-a8ef-ed3f23e48947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:43.986445 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e1a0b5-ffea-4df8-a8ef-ed3f23e48947 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:43.986445 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e1a0b5-ffea-4df8-a8ef-ed3f23e48947 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:43.987781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e1a0b5-ffea-4df8-a8ef-ed3f23e48947 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:43.989700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e1a0b5-ffea-4df8-a8ef-ed3f23e48947 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:43.990529 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e1a0b5-ffea-4df8-a8ef-ed3f23e48947 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:43.995458 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e1a0b5-ffea-4df8-a8ef-ed3f23e48947 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:44.018249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 186/373] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:44.023230 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e1a0b5-ffea-4df8-a8ef-ed3f23e48947 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:44.158535 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e1a0b5-ffea-4df8-a8ef-ed3f23e48947 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:44.182653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9d1b1e-a672-4520-a0a6-5d42d5abb14f None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:44.199470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bab961-f3f9-4f7d-9cae-54bc46e811a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:44.200695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bab961-f3f9-4f7d-9cae-54bc46e811a8 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:44.201324 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bab961-f3f9-4f7d-9cae-54bc46e811a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:44.201804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bab961-f3f9-4f7d-9cae-54bc46e811a8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:44.246366 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c296658-f51b-43f4-a57e-4c35bda919c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:44.247329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c296658-f51b-43f4-a57e-4c35bda919c7 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:44.247329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c296658-f51b-43f4-a57e-4c35bda919c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:44.247899 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c296658-f51b-43f4-a57e-4c35bda919c7 None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:44.254439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c296658-f51b-43f4-a57e-4c35bda919c7 None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:44.256100 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c296658-f51b-43f4-a57e-4c35bda919c7 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:44.257479 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:44.257613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 188/374] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:43 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:44.266017 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c296658-f51b-43f4-a57e-4c35bda919c7 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:44.276197 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:44.282117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6bab961-f3f9-4f7d-9cae-54bc46e811a8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:44.284004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6bab961-f3f9-4f7d-9cae-54bc46e811a8 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:44.407778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 187/375] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 04 00:46:44.410535 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c296658-f51b-43f4-a57e-4c35bda919c7 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:44.413606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bab961-f3f9-4f7d-9cae-54bc46e811a8 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:44.455139 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec12a061-10b2-4612-b4ff-3ae41ecad015 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:44.456059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec12a061-10b2-4612-b4ff-3ae41ecad015 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:44.457044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec12a061-10b2-4612-b4ff-3ae41ecad015 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:44.457563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec12a061-10b2-4612-b4ff-3ae41ecad015 None admin] PATH_INFO: `/v3/groups/bba303b549124f5898fa58d1e1f31db9` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:44.459998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec12a061-10b2-4612-b4ff-3ae41ecad015 None admin] RBAC: Authorizing `identity:get_group(group_id=bba303b549124f5898fa58d1e1f31db9)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:44.462889 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec12a061-10b2-4612-b4ff-3ae41ecad015 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:44.465284 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 189/376] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 4 00:46:44 2022] GET /identity/v3/groups/bba303b549124f5898fa58d1e1f31db9 => generated 226 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:44.471472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec12a061-10b2-4612-b4ff-3ae41ecad015 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:44.554642 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:44.569458 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:44.589990 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec12a061-10b2-4612-b4ff-3ae41ecad015 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:44.625704 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bab961-f3f9-4f7d-9cae-54bc46e811a8 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:44.642371 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d5a094-9ab0-4cfa-8df5-98d3e3cbcc3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:44.643013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d5a094-9ab0-4cfa-8df5-98d3e3cbcc3c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:44.643261 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d5a094-9ab0-4cfa-8df5-98d3e3cbcc3c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:44.643534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d5a094-9ab0-4cfa-8df5-98d3e3cbcc3c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:44.649142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d5a094-9ab0-4cfa-8df5-98d3e3cbcc3c None admin] RBAC: Adding query filter params (group.id=bba303b549124f5898fa58d1e1f31db9, role.id=c599b217579e42a49c657c642ea0648f, scope.project.id=d737f411ca514fb4aa93ee11d55926ae) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:44.649142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d5a094-9ab0-4cfa-8df5-98d3e3cbcc3c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:44.649142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d5a094-9ab0-4cfa-8df5-98d3e3cbcc3c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:44.661049 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 190/377] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 4 00:46:44 2022] GET /identity/v3/role_assignments?group.id=bba303b549124f5898fa58d1e1f31db9&scope.project.id=d737f411ca514fb4aa93ee11d55926ae&role.id=c599b217579e42a49c657c642ea0648f => generated 260 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:44.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbab885-8ec9-4750-9017-9c5a1ed11e8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:44.680823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbab885-8ec9-4750-9017-9c5a1ed11e8a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:44.681123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbab885-8ec9-4750-9017-9c5a1ed11e8a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:44.681408 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbab885-8ec9-4750-9017-9c5a1ed11e8a None admin] PATH_INFO: `/v3/projects/f25649add4b1426590abafbe586fc1fa` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:44.685965 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbab885-8ec9-4750-9017-9c5a1ed11e8a None admin] RBAC: Authorizing `identity:get_project(project_id=f25649add4b1426590abafbe586fc1fa)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:44.687461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcbab885-8ec9-4750-9017-9c5a1ed11e8a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:44.689389 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 188/378] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:44 2022] GET /identity/v3/projects/f25649add4b1426590abafbe586fc1fa => generated 344 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:44.695142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d5a094-9ab0-4cfa-8df5-98d3e3cbcc3c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:44.735295 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9e3e8d-3086-463c-a0f8-1b01a28a48ee None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:44.735597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9e3e8d-3086-463c-a0f8-1b01a28a48ee None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:44.736129 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9e3e8d-3086-463c-a0f8-1b01a28a48ee None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:44.737950 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 189/379] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:44.749103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c74449-2048-4b5c-8d2d-9fbced65cb10 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:44.749103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c74449-2048-4b5c-8d2d-9fbced65cb10 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:44.749103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c74449-2048-4b5c-8d2d-9fbced65cb10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:44.767549 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-d7c74449-2048-4b5c-8d2d-9fbced65cb10 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:44.769046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d7c74449-2048-4b5c-8d2d-9fbced65cb10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:44.885226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d5a094-9ab0-4cfa-8df5-98d3e3cbcc3c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:44.904053 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 190/380] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:46:44.935432 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6460a934-f5ea-4999-ba48-9a6652ca3c64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:44.936391 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6460a934-f5ea-4999-ba48-9a6652ca3c64 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:44.936898 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6460a934-f5ea-4999-ba48-9a6652ca3c64 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:44.937501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6460a934-f5ea-4999-ba48-9a6652ca3c64 None admin] PATH_INFO: `/v3/groups/d548703b5e7a4d17be559d2a5f32aa9c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:44.943202 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6460a934-f5ea-4999-ba48-9a6652ca3c64 None admin] RBAC: Authorizing `identity:get_group(group_id=d548703b5e7a4d17be559d2a5f32aa9c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:44.944980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6460a934-f5ea-4999-ba48-9a6652ca3c64 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:44.947281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 191/381] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 4 00:46:44 2022] GET /identity/v3/groups/d548703b5e7a4d17be559d2a5f32aa9c => generated 229 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:44.949912 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:44.953025 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c74449-2048-4b5c-8d2d-9fbced65cb10 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:44.983512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef892fc3-9abd-4b45-861b-0952288f3845 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:44.984256 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef892fc3-9abd-4b45-861b-0952288f3845 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:44.984749 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef892fc3-9abd-4b45-861b-0952288f3845 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:44.987420 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 192/382] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:44 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:46:44.993484 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0937c05-4db3-417d-a9b9-f6bcc7b373f0 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:44.994113 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0937c05-4db3-417d-a9b9-f6bcc7b373f0 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:44.994591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0937c05-4db3-417d-a9b9-f6bcc7b373f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:45.025482 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-a0937c05-4db3-417d-a9b9-f6bcc7b373f0 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:45.027872 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a0937c05-4db3-417d-a9b9-f6bcc7b373f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:45.086701 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:45.139659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c74449-2048-4b5c-8d2d-9fbced65cb10 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:45.172750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 193/383] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:45.196942 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6ef4ad-4f40-45b1-9635-207cac20d649 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:45.197499 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6ef4ad-4f40-45b1-9635-207cac20d649 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:45.197780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6ef4ad-4f40-45b1-9635-207cac20d649 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:45.198055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6ef4ad-4f40-45b1-9635-207cac20d649 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:45.198798 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6ef4ad-4f40-45b1-9635-207cac20d649 None admin] RBAC: Adding query filter params (group.id=d548703b5e7a4d17be559d2a5f32aa9c, role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=f25649add4b1426590abafbe586fc1fa) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:45.199289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6ef4ad-4f40-45b1-9635-207cac20d649 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:45.200738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6ef4ad-4f40-45b1-9635-207cac20d649 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:45.208356 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 191/384] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 4 00:46:44 2022] GET /identity/v3/role_assignments?group.id=d548703b5e7a4d17be559d2a5f32aa9c&scope.project.id=f25649add4b1426590abafbe586fc1fa&role.id=19c006412bec45dc8febd7cc86b9678f => generated 260 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:45.235343 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0937c05-4db3-417d-a9b9-f6bcc7b373f0 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:45.360689 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0937c05-4db3-417d-a9b9-f6bcc7b373f0 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:45.393688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:45.400526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc89a45d-6969-4ea5-86aa-1d126cd5e6ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:45.401212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc89a45d-6969-4ea5-86aa-1d126cd5e6ff None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:45.401496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc89a45d-6969-4ea5-86aa-1d126cd5e6ff None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:45.401771 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc89a45d-6969-4ea5-86aa-1d126cd5e6ff None admin] PATH_INFO: `/v3/roles/member` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:45.413739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc89a45d-6969-4ea5-86aa-1d126cd5e6ff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:45.420908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-fc89a45d-6969-4ea5-86aa-1d126cd5e6ff None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:45.420908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:45.420908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:45.420908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:45.420908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:45.420908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:45.420908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:45.420908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:45.420908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:45.420908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:45.420908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:45.420908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:45.420908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:45.420908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:45.420908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:45.420908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:45.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:45.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:45.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:45.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:45.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:45.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:45.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:45.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:45.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:45.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:45.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:45.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:45.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:45.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:45.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:45.422014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:45.423068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:45.423068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:45.423068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:45.423068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:45.423068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:45.423068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:45.423068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:45.423068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:45.423068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:45.423068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:45.423068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:45.423068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:45.423068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 194/385] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:45.430043 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6ef4ad-4f40-45b1-9635-207cac20d649 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:45.523005 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc89a45d-6969-4ea5-86aa-1d126cd5e6ff None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:45.573471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6ef4ad-4f40-45b1-9635-207cac20d649 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:45.641438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06947f78-e330-4e13-b460-bb258515b5da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:45.642431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06947f78-e330-4e13-b460-bb258515b5da None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:45.642912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06947f78-e330-4e13-b460-bb258515b5da None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:45.643439 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06947f78-e330-4e13-b460-bb258515b5da None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:45.644341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06947f78-e330-4e13-b460-bb258515b5da None admin] RBAC: Adding query filter params (name=member) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:45.645605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06947f78-e330-4e13-b460-bb258515b5da None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:45.647201 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06947f78-e330-4e13-b460-bb258515b5da None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:45.664549 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 192/386] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:45.671132 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06947f78-e330-4e13-b460-bb258515b5da None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:45.677814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:45.700160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc89a45d-6969-4ea5-86aa-1d126cd5e6ff None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:45.758756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32021165-ac80-474a-81a9-63ae77501db9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:45.759699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32021165-ac80-474a-81a9-63ae77501db9 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:45.760131 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32021165-ac80-474a-81a9-63ae77501db9 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:45.760552 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32021165-ac80-474a-81a9-63ae77501db9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:45.782796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32021165-ac80-474a-81a9-63ae77501db9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:45.793796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-32021165-ac80-474a-81a9-63ae77501db9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:45.793796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:45.793796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:45.793796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:45.793796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:45.793796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:45.793796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:45.793796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:45.793796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:45.793796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:45.793796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:45.793796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:45.793796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:45.793796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:45.793796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:45.793796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:45.795421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:45.795421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:45.795421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:45.795421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:45.795421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:45.795421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:45.795421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:45.795421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:45.795421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:45.795421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:45.795421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:45.795421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:45.795421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:45.795421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:45.795421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:45.795421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:45.796408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:45.796408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:45.796408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:45.796408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:45.796408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:45.796408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:45.796408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:45.796408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:45.796408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:45.796408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:45.796408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:45.796408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:45.798809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 195/387] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:46:45 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:46:45.807015 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32021165-ac80-474a-81a9-63ae77501db9 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:45.812405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06947f78-e330-4e13-b460-bb258515b5da None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:45.872154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d26b5c3-8265-4c20-9265-15b33f429bdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:45.873585 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d26b5c3-8265-4c20-9265-15b33f429bdc None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:45.874180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d26b5c3-8265-4c20-9265-15b33f429bdc None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:45.874713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d26b5c3-8265-4c20-9265-15b33f429bdc None admin] PATH_INFO: `/v3/groups/bba303b549124f5898fa58d1e1f31db9` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:45.881102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d26b5c3-8265-4c20-9265-15b33f429bdc None admin] RBAC: Authorizing `identity:get_group(group_id=bba303b549124f5898fa58d1e1f31db9)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:45.881102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d26b5c3-8265-4c20-9265-15b33f429bdc None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:45.882997 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 193/388] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 4 00:46:45 2022] GET /identity/v3/groups/bba303b549124f5898fa58d1e1f31db9 => generated 226 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:45.891194 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d26b5c3-8265-4c20-9265-15b33f429bdc None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:45.909123 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:46.001326 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:46.042047 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d26b5c3-8265-4c20-9265-15b33f429bdc None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:46.047548 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32021165-ac80-474a-81a9-63ae77501db9 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:46.085442 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4f4674-0c21-4d59-8c9e-d6472812cb05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:46.086269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4f4674-0c21-4d59-8c9e-d6472812cb05 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:46.086648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4f4674-0c21-4d59-8c9e-d6472812cb05 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:46.087023 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4f4674-0c21-4d59-8c9e-d6472812cb05 None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:46.088387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4f4674-0c21-4d59-8c9e-d6472812cb05 None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:46.096848 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4f4674-0c21-4d59-8c9e-d6472812cb05 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:46.096848 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 194/389] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:45 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:46.098610 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4f4674-0c21-4d59-8c9e-d6472812cb05 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:46.133151 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4efa2d0-e8ba-439f-92fb-1d331ed5ac62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:46.134240 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4efa2d0-e8ba-439f-92fb-1d331ed5ac62 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:46.134849 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4efa2d0-e8ba-439f-92fb-1d331ed5ac62 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:46.135322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4efa2d0-e8ba-439f-92fb-1d331ed5ac62 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:46.136197 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4efa2d0-e8ba-439f-92fb-1d331ed5ac62 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:46.136700 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4efa2d0-e8ba-439f-92fb-1d331ed5ac62 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:46.138470 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4efa2d0-e8ba-439f-92fb-1d331ed5ac62 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:46.157791 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 196/390] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:46:45 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 351 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:46.174270 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4efa2d0-e8ba-439f-92fb-1d331ed5ac62 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:46.233048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4f4674-0c21-4d59-8c9e-d6472812cb05 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:46.272132 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd75ac1a-1413-4b22-a623-3339426d3099 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:46.272968 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd75ac1a-1413-4b22-a623-3339426d3099 None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:46.281110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd75ac1a-1413-4b22-a623-3339426d3099 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:46.281110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd75ac1a-1413-4b22-a623-3339426d3099 None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a/groups/bba303b549124f5898fa58d1e1f31db9/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:46.281110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd75ac1a-1413-4b22-a623-3339426d3099 None admin] RBAC: Authorizing `identity:create_grant(project_id=1193eb16a76d432faf6141e47cc99c5a, group_id=bba303b549124f5898fa58d1e1f31db9, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:46.281110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd75ac1a-1413-4b22-a623-3339426d3099 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:46.375946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4efa2d0-e8ba-439f-92fb-1d331ed5ac62 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:46.381966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:46.389340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 195/391] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Sep 4 00:46:46 2022] PUT /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a/groups/bba303b549124f5898fa58d1e1f31db9/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:46:46.390913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133eb0f3-1ba6-411b-8ae0-35c0308e314c None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:46.390913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133eb0f3-1ba6-411b-8ae0-35c0308e314c None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:46.390913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133eb0f3-1ba6-411b-8ae0-35c0308e314c None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:46.392242 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 196/392] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:46 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:46.399015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03809837-25db-41ab-94d0-345fa2db6ca2 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:46.399349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03809837-25db-41ab-94d0-345fa2db6ca2 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:46.399606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03809837-25db-41ab-94d0-345fa2db6ca2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:46.401709 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:46.447995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-03809837-25db-41ab-94d0-345fa2db6ca2 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:46.449285 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-03809837-25db-41ab-94d0-345fa2db6ca2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:46.489836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f23be6f-ba0e-4f34-a451-78faad06c497 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:46.490534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f23be6f-ba0e-4f34-a451-78faad06c497 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:46.490957 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f23be6f-ba0e-4f34-a451-78faad06c497 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:46.491078 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f23be6f-ba0e-4f34-a451-78faad06c497 None admin] PATH_INFO: `/v3/groups/bba303b549124f5898fa58d1e1f31db9` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:46.493027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f23be6f-ba0e-4f34-a451-78faad06c497 None admin] RBAC: Authorizing `identity:get_group(group_id=bba303b549124f5898fa58d1e1f31db9)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:46.494548 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f23be6f-ba0e-4f34-a451-78faad06c497 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:46.502950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 197/393] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 4 00:46:46 2022] GET /identity/v3/groups/bba303b549124f5898fa58d1e1f31db9 => generated 226 bytes in 329 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:46.504571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f23be6f-ba0e-4f34-a451-78faad06c497 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:46.599013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 197/394] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:46:46.723353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f23be6f-ba0e-4f34-a451-78faad06c497 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:46.766979 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fcb401-1ca5-460d-b7df-48ef835a437b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:46.767623 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fcb401-1ca5-460d-b7df-48ef835a437b None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:46.767904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fcb401-1ca5-460d-b7df-48ef835a437b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:46.768181 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fcb401-1ca5-460d-b7df-48ef835a437b None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:46.769888 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83fcb401-1ca5-460d-b7df-48ef835a437b None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:46.771635 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83fcb401-1ca5-460d-b7df-48ef835a437b None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:46.773590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 198/395] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:46 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:46.780757 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03809837-25db-41ab-94d0-345fa2db6ca2 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:46.896263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03809837-25db-41ab-94d0-345fa2db6ca2 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:46.929455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:46.936243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44f374e-6a38-43c3-ab07-da853ece75de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:46.941868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f374e-6a38-43c3-ab07-da853ece75de None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:46.941868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f374e-6a38-43c3-ab07-da853ece75de None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:46.941868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f374e-6a38-43c3-ab07-da853ece75de None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a/groups/bba303b549124f5898fa58d1e1f31db9/roles/c599b217579e42a49c657c642ea0648f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:46.941868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44f374e-6a38-43c3-ab07-da853ece75de None admin] RBAC: Authorizing `identity:create_grant(project_id=1193eb16a76d432faf6141e47cc99c5a, group_id=bba303b549124f5898fa58d1e1f31db9, role_id=c599b217579e42a49c657c642ea0648f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:46.941868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44f374e-6a38-43c3-ab07-da853ece75de None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:46.998037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fcb401-1ca5-460d-b7df-48ef835a437b None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:47.018212 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 198/396] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Sep 4 00:46:46 2022] PUT /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a/groups/bba303b549124f5898fa58d1e1f31db9/roles/c599b217579e42a49c657c642ea0648f => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:46:47.069780 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:47.134150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fcb401-1ca5-460d-b7df-48ef835a437b None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:47.178208 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cf9af8-f213-4c15-b888-7c208333863d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:47.178859 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cf9af8-f213-4c15-b888-7c208333863d None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:47.179141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cf9af8-f213-4c15-b888-7c208333863d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:47.179419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cf9af8-f213-4c15-b888-7c208333863d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:47.194032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90cf9af8-f213-4c15-b888-7c208333863d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:47.199678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-90cf9af8-f213-4c15-b888-7c208333863d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:47.199678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:47.199678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:47.199678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:47.199678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:47.199678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:47.199678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:47.199678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:47.199678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:47.199678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:47.199678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:47.199678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:47.199678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:47.199678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:47.199678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:47.199678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:47.200613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:47.200613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:47.200613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:47.200613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:47.200613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:47.200613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:47.200613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:47.200613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:47.200613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:47.200613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:47.200613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:47.200613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:47.200613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:47.200613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:47.200613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:47.200613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:47.201457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:47.201457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:47.201457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:47.201457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:47.201457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:47.201457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:47.201457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:47.201457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:47.201457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:47.201457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:47.201457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:47.201457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:47.201457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 199/397] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:46 2022] GET /identity/v3/roles/member => generated 84 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:47.207461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44f374e-6a38-43c3-ab07-da853ece75de None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:47.364353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44f374e-6a38-43c3-ab07-da853ece75de None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:47.392251 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f232a7-ba82-4503-afe9-cb4d1e6f783a None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:47.392984 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f232a7-ba82-4503-afe9-cb4d1e6f783a None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:47.393604 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f232a7-ba82-4503-afe9-cb4d1e6f783a None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:47.395468 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 200/398] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:47.400972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9359a06-5077-4ea7-b869-245222a21850 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:47.401549 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9359a06-5077-4ea7-b869-245222a21850 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:47.402037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9359a06-5077-4ea7-b869-245222a21850 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:47.409192 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a404c623-65c9-42d0-b913-454c48810b8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:47.409192 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a404c623-65c9-42d0-b913-454c48810b8d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:47.409192 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a404c623-65c9-42d0-b913-454c48810b8d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:47.409192 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a404c623-65c9-42d0-b913-454c48810b8d None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:47.409551 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a404c623-65c9-42d0-b913-454c48810b8d None admin] RBAC: Adding query filter params (name=member) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:47.409551 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a404c623-65c9-42d0-b913-454c48810b8d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:47.410795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a404c623-65c9-42d0-b913-454c48810b8d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:47.420056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 199/399] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:47.429787 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a404c623-65c9-42d0-b913-454c48810b8d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:47.437370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-b9359a06-5077-4ea7-b869-245222a21850 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:47.439006 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b9359a06-5077-4ea7-b869-245222a21850 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:47.490186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:47.545313 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a404c623-65c9-42d0-b913-454c48810b8d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:47.578561 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 201/400] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:46:47.582719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975f2a3d-2c59-4fba-8d37-4346bd69c4cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:47.583382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975f2a3d-2c59-4fba-8d37-4346bd69c4cd None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:47.583864 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975f2a3d-2c59-4fba-8d37-4346bd69c4cd None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:47.584133 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975f2a3d-2c59-4fba-8d37-4346bd69c4cd None admin] PATH_INFO: `/v3/groups/bba303b549124f5898fa58d1e1f31db9` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:47.585290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975f2a3d-2c59-4fba-8d37-4346bd69c4cd None admin] RBAC: Authorizing `identity:get_group(group_id=bba303b549124f5898fa58d1e1f31db9)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:47.586707 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975f2a3d-2c59-4fba-8d37-4346bd69c4cd None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:47.588560 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 200/401] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 4 00:46:47 2022] GET /identity/v3/groups/bba303b549124f5898fa58d1e1f31db9 => generated 226 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:47.595739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9359a06-5077-4ea7-b869-245222a21850 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:47.661554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:47.734648 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9359a06-5077-4ea7-b869-245222a21850 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:47.783870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a7f93c-b135-4dc7-ba8c-b0ca6c1942a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:47.785462 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a7f93c-b135-4dc7-ba8c-b0ca6c1942a3 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:47.786117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a7f93c-b135-4dc7-ba8c-b0ca6c1942a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:47.786598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a7f93c-b135-4dc7-ba8c-b0ca6c1942a3 None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:47.793856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a7f93c-b135-4dc7-ba8c-b0ca6c1942a3 None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:47.796262 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a7f93c-b135-4dc7-ba8c-b0ca6c1942a3 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:47.798226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 202/402] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:47 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:47.804936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975f2a3d-2c59-4fba-8d37-4346bd69c4cd None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:47.940004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975f2a3d-2c59-4fba-8d37-4346bd69c4cd None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:47.964966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:47.987226 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef038794-f78e-45a4-a443-4bd066f8e1a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:47.987908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef038794-f78e-45a4-a443-4bd066f8e1a8 None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:47.988354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef038794-f78e-45a4-a443-4bd066f8e1a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:47.988669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef038794-f78e-45a4-a443-4bd066f8e1a8 None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae/groups/bba303b549124f5898fa58d1e1f31db9/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:47.993839 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef038794-f78e-45a4-a443-4bd066f8e1a8 None admin] RBAC: Authorizing `identity:create_grant(project_id=d737f411ca514fb4aa93ee11d55926ae, group_id=bba303b549124f5898fa58d1e1f31db9, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:47.994656 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a7f93c-b135-4dc7-ba8c-b0ca6c1942a3 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:47.995285 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef038794-f78e-45a4-a443-4bd066f8e1a8 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:48.106966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 201/403] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Sep 4 00:46:47 2022] PUT /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae/groups/bba303b549124f5898fa58d1e1f31db9/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:46:48.130865 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a7f93c-b135-4dc7-ba8c-b0ca6c1942a3 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:48.139154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:48.170296 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef742ce6-9bed-42ee-bb5a-2a1013a8c919 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:48.171106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef742ce6-9bed-42ee-bb5a-2a1013a8c919 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:48.171388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef742ce6-9bed-42ee-bb5a-2a1013a8c919 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:48.171729 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef742ce6-9bed-42ee-bb5a-2a1013a8c919 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:48.184520 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef742ce6-9bed-42ee-bb5a-2a1013a8c919 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:48.189526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-ef742ce6-9bed-42ee-bb5a-2a1013a8c919 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:48.189526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:48.189526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:48.189526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:48.189526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:48.189526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:48.189526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:48.189526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:48.189526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:48.189526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:48.189526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:48.189526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:48.189526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:48.189526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:48.189526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:48.189526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:48.190486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:48.190486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:48.190486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:48.190486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:48.190486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:48.190486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:48.190486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:48.190486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:48.190486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:48.190486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:48.190486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:48.190486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:48.190486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:48.190486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:48.190486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:48.190486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:48.192104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:48.192104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:48.192104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:48.192104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:48.192104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:48.192104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:48.192104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:48.192104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:48.192104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:48.192104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:48.192104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:48.192104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:48.192104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 203/404] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:46:47 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:48.199618 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef038794-f78e-45a4-a443-4bd066f8e1a8 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:48.224506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4af19b-8d24-47d7-9013-581492b55fdc None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:48.224506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4af19b-8d24-47d7-9013-581492b55fdc None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:48.224794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4af19b-8d24-47d7-9013-581492b55fdc None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:48.228283 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 204/405] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:46:48.231590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849597ff-7c8a-463d-a03e-ac4393b9e01d None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:48.231866 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849597ff-7c8a-463d-a03e-ac4393b9e01d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:48.232137 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849597ff-7c8a-463d-a03e-ac4393b9e01d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:48.258365 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-849597ff-7c8a-463d-a03e-ac4393b9e01d None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:48.259711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-849597ff-7c8a-463d-a03e-ac4393b9e01d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:48.331214 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef038794-f78e-45a4-a443-4bd066f8e1a8 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:48.338619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 205/406] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:46:48.368743 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8894c273-4349-4506-94b5-ad3d803f995c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:48.369662 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8894c273-4349-4506-94b5-ad3d803f995c None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:48.370091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8894c273-4349-4506-94b5-ad3d803f995c None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:48.370533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8894c273-4349-4506-94b5-ad3d803f995c None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:48.371407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8894c273-4349-4506-94b5-ad3d803f995c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:48.371871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8894c273-4349-4506-94b5-ad3d803f995c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:48.373824 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8894c273-4349-4506-94b5-ad3d803f995c None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:48.383383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 202/407] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:46:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:48.389842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849597ff-7c8a-463d-a03e-ac4393b9e01d None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:48.511668 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:48.517950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849597ff-7c8a-463d-a03e-ac4393b9e01d None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:48.558828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51014892-14b4-49e3-a97f-ee3288615e4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:48.559846 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51014892-14b4-49e3-a97f-ee3288615e4c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:48.560334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51014892-14b4-49e3-a97f-ee3288615e4c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:48.560827 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51014892-14b4-49e3-a97f-ee3288615e4c None admin] PATH_INFO: `/v3/groups/bba303b549124f5898fa58d1e1f31db9` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:48.562226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51014892-14b4-49e3-a97f-ee3288615e4c None admin] RBAC: Authorizing `identity:get_group(group_id=bba303b549124f5898fa58d1e1f31db9)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:48.563870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51014892-14b4-49e3-a97f-ee3288615e4c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:48.565793 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 206/408] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 4 00:46:48 2022] GET /identity/v3/groups/bba303b549124f5898fa58d1e1f31db9 => generated 226 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:48.572311 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8894c273-4349-4506-94b5-ad3d803f995c None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:48.621515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:48.694112 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8894c273-4349-4506-94b5-ad3d803f995c None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:48.736529 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0190a70-5042-418f-9c88-bd04db63cb57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:48.737112 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0190a70-5042-418f-9c88-bd04db63cb57 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:48.737406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0190a70-5042-418f-9c88-bd04db63cb57 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:48.737688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0190a70-5042-418f-9c88-bd04db63cb57 None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:48.739940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0190a70-5042-418f-9c88-bd04db63cb57 None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:48.741521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0190a70-5042-418f-9c88-bd04db63cb57 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:48.744074 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51014892-14b4-49e3-a97f-ee3288615e4c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:48.745327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 203/409] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:48 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:48.751586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0190a70-5042-418f-9c88-bd04db63cb57 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:48.864759 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0190a70-5042-418f-9c88-bd04db63cb57 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:48.891651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:48.902455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2062fc-2900-4c8c-9928-6ef75d291eea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:48.903213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2062fc-2900-4c8c-9928-6ef75d291eea None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:48.903409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2062fc-2900-4c8c-9928-6ef75d291eea None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:48.903682 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2062fc-2900-4c8c-9928-6ef75d291eea None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae/groups/bba303b549124f5898fa58d1e1f31db9/roles/c599b217579e42a49c657c642ea0648f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:48.905976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2062fc-2900-4c8c-9928-6ef75d291eea None admin] RBAC: Authorizing `identity:create_grant(project_id=d737f411ca514fb4aa93ee11d55926ae, group_id=bba303b549124f5898fa58d1e1f31db9, role_id=c599b217579e42a49c657c642ea0648f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:48.907415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2062fc-2900-4c8c-9928-6ef75d291eea None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:48.938224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51014892-14b4-49e3-a97f-ee3288615e4c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:48.977177 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c26a61-4e8d-4db5-9789-9e7934701bee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:48.977880 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c26a61-4e8d-4db5-9789-9e7934701bee None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:48.978164 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c26a61-4e8d-4db5-9789-9e7934701bee None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:48.978435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c26a61-4e8d-4db5-9789-9e7934701bee None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:48.989897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c26a61-4e8d-4db5-9789-9e7934701bee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:49.000437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 204/410] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Sep 4 00:46:48 2022] PUT /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae/groups/bba303b549124f5898fa58d1e1f31db9/roles/c599b217579e42a49c657c642ea0648f => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 00:46:49.009770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-05c26a61-4e8d-4db5-9789-9e7934701bee None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:49.009770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:49.009770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:49.009770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:49.009770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:49.009770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:49.009770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:49.009770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:49.009770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:49.009770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:49.009770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:49.009770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:49.009770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:49.009770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:49.009770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:49.009770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:49.010808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:49.010808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:49.010808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:49.010808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:49.010808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:49.010808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:49.010808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:49.010808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:49.010808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:49.010808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:49.010808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:49.010808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:49.010808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:49.010808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:49.010808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:49.010808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:49.012042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:49.012042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:49.012042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:49.012042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:49.012042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:49.012042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:49.012042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:49.012042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:49.012042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:49.012042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:49.012042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:49.012042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:49.012042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 207/411] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:46:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:46:49.019126 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2062fc-2900-4c8c-9928-6ef75d291eea None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:49.115599 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57441ff0-831b-453b-8569-76cc5768d314 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:49.115945 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57441ff0-831b-453b-8569-76cc5768d314 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:49.116153 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57441ff0-831b-453b-8569-76cc5768d314 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:49.117566 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 208/412] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:46:49.123464 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8a168f-f4bd-44f7-9838-3d76d2e4465d None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:49.123757 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8a168f-f4bd-44f7-9838-3d76d2e4465d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:49.124017 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8a168f-f4bd-44f7-9838-3d76d2e4465d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:49.140668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2062fc-2900-4c8c-9928-6ef75d291eea None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:49.166295 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-5b8a168f-f4bd-44f7-9838-3d76d2e4465d None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:49.167548 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-5b8a168f-f4bd-44f7-9838-3d76d2e4465d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:49.181104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56486c38-70cb-4c40-a6ab-83a473cdfd57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:49.181818 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56486c38-70cb-4c40-a6ab-83a473cdfd57 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:49.182103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56486c38-70cb-4c40-a6ab-83a473cdfd57 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:49.182379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56486c38-70cb-4c40-a6ab-83a473cdfd57 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:49.183123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56486c38-70cb-4c40-a6ab-83a473cdfd57 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:49.183611 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56486c38-70cb-4c40-a6ab-83a473cdfd57 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:49.185068 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56486c38-70cb-4c40-a6ab-83a473cdfd57 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:49.195769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 205/413] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:49.202183 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56486c38-70cb-4c40-a6ab-83a473cdfd57 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:49.206470 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:49.240935 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:49.277525 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 209/414] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:49.329104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56486c38-70cb-4c40-a6ab-83a473cdfd57 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:49.380984 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec0eb5a-8b02-458a-a83e-289eb522b2ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:49.382023 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec0eb5a-8b02-458a-a83e-289eb522b2ed None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:49.382510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec0eb5a-8b02-458a-a83e-289eb522b2ed None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:49.383002 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec0eb5a-8b02-458a-a83e-289eb522b2ed None admin] PATH_INFO: `/v3/groups/d548703b5e7a4d17be559d2a5f32aa9c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:49.384471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec0eb5a-8b02-458a-a83e-289eb522b2ed None admin] RBAC: Authorizing `identity:get_group(group_id=d548703b5e7a4d17be559d2a5f32aa9c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:49.386313 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec0eb5a-8b02-458a-a83e-289eb522b2ed None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:49.388131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 206/415] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 4 00:46:49 2022] GET /identity/v3/groups/d548703b5e7a4d17be559d2a5f32aa9c => generated 229 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:49.393965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8a168f-f4bd-44f7-9838-3d76d2e4465d None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:49.503124 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8a168f-f4bd-44f7-9838-3d76d2e4465d None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:49.540857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbd5a21-4379-427e-b8d6-8bed459c9738 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:49.541575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbd5a21-4379-427e-b8d6-8bed459c9738 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:49.541866 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbd5a21-4379-427e-b8d6-8bed459c9738 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:49.542147 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbd5a21-4379-427e-b8d6-8bed459c9738 None admin] PATH_INFO: `/v3/projects/f25649add4b1426590abafbe586fc1fa` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:49.544741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbd5a21-4379-427e-b8d6-8bed459c9738 None admin] RBAC: Authorizing `identity:get_project(project_id=f25649add4b1426590abafbe586fc1fa)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:49.547046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbd5a21-4379-427e-b8d6-8bed459c9738 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:49.553192 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 210/416] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:49 2022] GET /identity/v3/projects/f25649add4b1426590abafbe586fc1fa => generated 344 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:49.559471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec0eb5a-8b02-458a-a83e-289eb522b2ed None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:49.691423 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:49.695418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec0eb5a-8b02-458a-a83e-289eb522b2ed None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:49.704970 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbd5a21-4379-427e-b8d6-8bed459c9738 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:49.735542 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914aeaed-7428-4c52-8f72-b37bed000061 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:49.736202 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914aeaed-7428-4c52-8f72-b37bed000061 None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:49.736480 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914aeaed-7428-4c52-8f72-b37bed000061 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:49.736752 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914aeaed-7428-4c52-8f72-b37bed000061 None admin] PATH_INFO: `/v3/projects/f25649add4b1426590abafbe586fc1fa/groups/d548703b5e7a4d17be559d2a5f32aa9c/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:49.739269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914aeaed-7428-4c52-8f72-b37bed000061 None admin] RBAC: Authorizing `identity:create_grant(project_id=f25649add4b1426590abafbe586fc1fa, group_id=d548703b5e7a4d17be559d2a5f32aa9c, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:49.740849 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914aeaed-7428-4c52-8f72-b37bed000061 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:49.780050 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:49.821658 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 207/417] 10.222.0.42 () {58 vars in 1442 bytes} [Sun Sep 4 00:46:49 2022] PUT /identity/v3/projects/f25649add4b1426590abafbe586fc1fa/groups/d548703b5e7a4d17be559d2a5f32aa9c/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:46:49.852590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbd5a21-4379-427e-b8d6-8bed459c9738 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:49.901512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a37ec12-5cd7-4b88-9b60-04695eb84d16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:49.902169 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a37ec12-5cd7-4b88-9b60-04695eb84d16 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:49.902456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a37ec12-5cd7-4b88-9b60-04695eb84d16 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:49.902731 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a37ec12-5cd7-4b88-9b60-04695eb84d16 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:49.915648 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a37ec12-5cd7-4b88-9b60-04695eb84d16 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:49.922577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-6a37ec12-5cd7-4b88-9b60-04695eb84d16 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:49.922577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:49.922577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:49.922577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:49.922577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:49.922577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:49.922577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:49.922577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:49.922577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:49.922577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:49.922577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:49.922577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:49.922577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:49.922577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:49.922577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:49.922577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:49.923511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:49.923511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:49.923511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:49.923511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:49.923511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:49.923511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:49.923511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:49.923511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:49.923511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:49.923511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:49.923511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:49.923511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:49.923511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:49.923511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:49.923511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:49.923511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:49.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:49.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:49.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:49.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:49.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:49.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:49.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:49.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:49.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:49.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:49.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:49.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:49.924319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 211/418] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:46:49.934603 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914aeaed-7428-4c52-8f72-b37bed000061 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:50.065011 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f91677-0b6f-4a3a-8361-516b85989ff8 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:50.065011 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f91677-0b6f-4a3a-8361-516b85989ff8 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:50.065011 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f91677-0b6f-4a3a-8361-516b85989ff8 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:50.065011 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 212/419] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:50.072076 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a968fba-4dcc-4001-ae83-9fb6f9eee5e3 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:50.072362 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a968fba-4dcc-4001-ae83-9fb6f9eee5e3 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:50.072634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a968fba-4dcc-4001-ae83-9fb6f9eee5e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:50.082199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914aeaed-7428-4c52-8f72-b37bed000061 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:50.092492 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-4a968fba-4dcc-4001-ae83-9fb6f9eee5e3 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:50.093698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4a968fba-4dcc-4001-ae83-9fb6f9eee5e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:50.126830 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a34210f-750d-4b13-b1b6-fefd17200629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:50.127442 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a34210f-750d-4b13-b1b6-fefd17200629 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:50.127720 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a34210f-750d-4b13-b1b6-fefd17200629 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:50.127994 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a34210f-750d-4b13-b1b6-fefd17200629 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:50.128714 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a34210f-750d-4b13-b1b6-fefd17200629 None admin] RBAC: Adding query filter params (name=member) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:50.129020 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a34210f-750d-4b13-b1b6-fefd17200629 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:50.130412 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a34210f-750d-4b13-b1b6-fefd17200629 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:50.143556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 208/420] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:50.153043 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a34210f-750d-4b13-b1b6-fefd17200629 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:50.193339 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 213/421] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:46:50.196838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:50.289046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a34210f-750d-4b13-b1b6-fefd17200629 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:50.342669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20454af5-7521-4cc4-8689-e829eefff7a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:50.343207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20454af5-7521-4cc4-8689-e829eefff7a6 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:50.343489 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20454af5-7521-4cc4-8689-e829eefff7a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:50.343761 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20454af5-7521-4cc4-8689-e829eefff7a6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:50.429244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20454af5-7521-4cc4-8689-e829eefff7a6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:50.434391 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20454af5-7521-4cc4-8689-e829eefff7a6 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:50.453793 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:50.538531 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 209/422] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:50.544181 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a968fba-4dcc-4001-ae83-9fb6f9eee5e3 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:50.569401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20454af5-7521-4cc4-8689-e829eefff7a6 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:50.574604 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:50.695518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20454af5-7521-4cc4-8689-e829eefff7a6 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:50.716407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:50.738797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05c20f5-6812-4695-87e8-e1fc6561f227 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:50.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05c20f5-6812-4695-87e8-e1fc6561f227 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:50.739818 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05c20f5-6812-4695-87e8-e1fc6561f227 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:50.740092 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05c20f5-6812-4695-87e8-e1fc6561f227 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:50.765312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a968fba-4dcc-4001-ae83-9fb6f9eee5e3 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:50.767248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05c20f5-6812-4695-87e8-e1fc6561f227 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:50.772216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-b05c20f5-6812-4695-87e8-e1fc6561f227 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:50.772216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:50.772216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:50.772216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:50.772216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:50.772216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:50.772216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:50.772216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:50.772216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:50.772216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:50.772216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:50.772216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:50.772216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:50.772216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:50.772216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:50.772216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:50.773117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:50.773117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:50.773117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:50.773117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:50.773117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:50.773117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:50.773117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:50.773117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:50.773117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:50.773117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:50.773117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:50.773117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:50.773117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:50.773117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:50.773117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:50.773117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:50.773928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:50.773928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:50.773928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:50.773928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:50.773928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:50.773928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:50.773928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:50.773928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:50.773928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:50.773928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:50.773928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:50.773928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:50.774527 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 210/423] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:46:50 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:46:50.781511 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05c20f5-6812-4695-87e8-e1fc6561f227 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:50.824873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03b08e6-ba57-43cb-a61c-63f38e68425b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:50.825545 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03b08e6-ba57-43cb-a61c-63f38e68425b None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:50.825822 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03b08e6-ba57-43cb-a61c-63f38e68425b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:50.826094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03b08e6-ba57-43cb-a61c-63f38e68425b None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:50.827267 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03b08e6-ba57-43cb-a61c-63f38e68425b None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:50.828814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03b08e6-ba57-43cb-a61c-63f38e68425b None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:50.830679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 214/424] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:50 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:50.831948 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b824f0-405d-474c-877f-1e880f2c8dcc None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:50.832429 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b824f0-405d-474c-877f-1e880f2c8dcc None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:50.832711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b824f0-405d-474c-877f-1e880f2c8dcc None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:50.838136 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 215/425] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:50.838904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b824f0-405d-474c-877f-1e880f2c8dcc None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:50.906819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05c20f5-6812-4695-87e8-e1fc6561f227 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:50.946853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cfef33-ae85-4d82-8e61-e538ade69f83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:50.947383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cfef33-ae85-4d82-8e61-e538ade69f83 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:50.947667 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cfef33-ae85-4d82-8e61-e538ade69f83 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:50.947942 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cfef33-ae85-4d82-8e61-e538ade69f83 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:50.948617 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cfef33-ae85-4d82-8e61-e538ade69f83 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:50.948923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cfef33-ae85-4d82-8e61-e538ade69f83 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:50.950330 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:50.950508 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cfef33-ae85-4d82-8e61-e538ade69f83 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:50.960554 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 211/426] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:46:50 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:50.962056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975cb16e-642f-4613-972b-e2a017b898a2 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:50.962348 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975cb16e-642f-4613-972b-e2a017b898a2 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:50.962622 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975cb16e-642f-4613-972b-e2a017b898a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:50.981910 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b824f0-405d-474c-877f-1e880f2c8dcc None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:50.983502 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-975cb16e-642f-4613-972b-e2a017b898a2 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:50.984834 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-975cb16e-642f-4613-972b-e2a017b898a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:51.025410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:51.029002 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62be7e28-79e4-4029-857c-910bc5e58bf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:51.029839 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62be7e28-79e4-4029-857c-910bc5e58bf5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:51.030262 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62be7e28-79e4-4029-857c-910bc5e58bf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:51.030677 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62be7e28-79e4-4029-857c-910bc5e58bf5 None admin] PATH_INFO: `/v3/groups/bba303b549124f5898fa58d1e1f31db9` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:51.032388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62be7e28-79e4-4029-857c-910bc5e58bf5 None admin] RBAC: Authorizing `identity:get_group(group_id=bba303b549124f5898fa58d1e1f31db9)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:51.034338 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62be7e28-79e4-4029-857c-910bc5e58bf5 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:51.037656 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 216/427] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 4 00:46:50 2022] GET /identity/v3/groups/bba303b549124f5898fa58d1e1f31db9 => generated 226 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:51.038858 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62be7e28-79e4-4029-857c-910bc5e58bf5 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:51.065100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 212/428] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:51.065100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975cb16e-642f-4613-972b-e2a017b898a2 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:51.194883 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62be7e28-79e4-4029-857c-910bc5e58bf5 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:51.236679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3c2a03-2bfa-4576-9010-d435ef65fb9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:51.237225 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c2a03-2bfa-4576-9010-d435ef65fb9d None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:51.237501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c2a03-2bfa-4576-9010-d435ef65fb9d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:51.237767 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c2a03-2bfa-4576-9010-d435ef65fb9d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:51.256562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975cb16e-642f-4613-972b-e2a017b898a2 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:51.289758 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:51.300951 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60d2bc4-3118-44b7-aec4-1950a4fc1885 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:51.301994 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60d2bc4-3118-44b7-aec4-1950a4fc1885 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:51.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60d2bc4-3118-44b7-aec4-1950a4fc1885 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:51.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60d2bc4-3118-44b7-aec4-1950a4fc1885 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:51.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60d2bc4-3118-44b7-aec4-1950a4fc1885 None admin] RBAC: Adding query filter params (group.id=bba303b549124f5898fa58d1e1f31db9, role.id=4bfd6d0544ee4a9e933538e78135a12c, scope.project.id=1193eb16a76d432faf6141e47cc99c5a) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:51.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60d2bc4-3118-44b7-aec4-1950a4fc1885 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:51.313011 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b60d2bc4-3118-44b7-aec4-1950a4fc1885 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:51.313972 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 213/429] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 4 00:46:51 2022] GET /identity/v3/role_assignments?group.id=bba303b549124f5898fa58d1e1f31db9&scope.project.id=1193eb16a76d432faf6141e47cc99c5a&role.id=4bfd6d0544ee4a9e933538e78135a12c => generated 611 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:51.362968 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33c9beb-e3c2-4cd1-8736-6beea90b20a3 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:51.363836 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33c9beb-e3c2-4cd1-8736-6beea90b20a3 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:51.364067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33c9beb-e3c2-4cd1-8736-6beea90b20a3 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:51.369894 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 214/430] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:51 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:46:51.378609 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9cb468-6419-4add-aa2b-4a235c4e4f4f None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:51.381796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9cb468-6419-4add-aa2b-4a235c4e4f4f None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:51.381796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9cb468-6419-4add-aa2b-4a235c4e4f4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:51.388081 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3c2a03-2bfa-4576-9010-d435ef65fb9d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:51.390037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3c2a03-2bfa-4576-9010-d435ef65fb9d None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:51.427012 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-fe9cb468-6419-4add-aa2b-4a235c4e4f4f None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:51.428268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-fe9cb468-6419-4add-aa2b-4a235c4e4f4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:51.475413 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:51.525597 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 217/431] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 487 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:51.526409 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3c2a03-2bfa-4576-9010-d435ef65fb9d None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:51.533552 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 215/432] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:51.533552 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9cb468-6419-4add-aa2b-4a235c4e4f4f None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:51.586133 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:51.646496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3c2a03-2bfa-4576-9010-d435ef65fb9d None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:51.658911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9cb468-6419-4add-aa2b-4a235c4e4f4f None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:51.692353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc398f89-57ab-4d21-a5ea-e5463b08a6e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:51.693120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc398f89-57ab-4d21-a5ea-e5463b08a6e2 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:51.693469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc398f89-57ab-4d21-a5ea-e5463b08a6e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:51.693807 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc398f89-57ab-4d21-a5ea-e5463b08a6e2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:51.710080 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc398f89-57ab-4d21-a5ea-e5463b08a6e2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:51.714841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-cc398f89-57ab-4d21-a5ea-e5463b08a6e2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:51.714841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:51.714841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:51.714841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:51.714841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:51.714841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:51.714841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:51.714841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:51.714841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:51.714841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:51.714841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:51.714841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:51.714841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:51.714841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:51.714841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:51.714841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:51.715699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:51.715699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:51.715699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:51.715699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:51.715699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:51.715699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:51.715699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:51.715699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:51.715699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:51.715699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:51.715699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:51.715699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:51.715699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:51.715699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:51.715699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:51.715699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:51.716578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:51.716578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:51.716578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:51.716578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:51.716578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:51.716578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:51.716578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:51.716578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:51.716578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:51.716578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:51.716578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 04 00:46:51.716578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:51.716578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 218/433] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:51.719416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d507818f-c3b8-42b3-838e-9eabee6bc7be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:51.720072 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d507818f-c3b8-42b3-838e-9eabee6bc7be None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:51.720356 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d507818f-c3b8-42b3-838e-9eabee6bc7be None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:51.720909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d507818f-c3b8-42b3-838e-9eabee6bc7be None admin] PATH_INFO: `/v3/projects/1193eb16a76d432faf6141e47cc99c5a` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:51.721768 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc398f89-57ab-4d21-a5ea-e5463b08a6e2 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:51.722160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d507818f-c3b8-42b3-838e-9eabee6bc7be None admin] RBAC: Authorizing `identity:get_project(project_id=1193eb16a76d432faf6141e47cc99c5a)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:51.724369 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d507818f-c3b8-42b3-838e-9eabee6bc7be None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:51.726467 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 216/434] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:51 2022] GET /identity/v3/projects/1193eb16a76d432faf6141e47cc99c5a => generated 298 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:51.732336 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d507818f-c3b8-42b3-838e-9eabee6bc7be None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:51.776091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:51.832487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc398f89-57ab-4d21-a5ea-e5463b08a6e2 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:51.842493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:51.849147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d507818f-c3b8-42b3-838e-9eabee6bc7be None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:51.870156 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff46310-6010-44aa-b341-5cb4b5c22f42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:51.870997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff46310-6010-44aa-b341-5cb4b5c22f42 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:51.871413 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff46310-6010-44aa-b341-5cb4b5c22f42 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:51.871854 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff46310-6010-44aa-b341-5cb4b5c22f42 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:51.872990 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff46310-6010-44aa-b341-5cb4b5c22f42 None admin] RBAC: Adding query filter params (name=member) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:51.873478 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff46310-6010-44aa-b341-5cb4b5c22f42 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:51.874999 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff46310-6010-44aa-b341-5cb4b5c22f42 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:51.885174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 219/435] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:46:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:51.891355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff46310-6010-44aa-b341-5cb4b5c22f42 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:51.902549 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00107eff-3b29-4121-aeae-30c5203403f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:51.903248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00107eff-3b29-4121-aeae-30c5203403f1 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:51.903532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00107eff-3b29-4121-aeae-30c5203403f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:51.903855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00107eff-3b29-4121-aeae-30c5203403f1 None admin] PATH_INFO: `/v3/groups/bba303b549124f5898fa58d1e1f31db9` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:51.905499 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00107eff-3b29-4121-aeae-30c5203403f1 None admin] RBAC: Authorizing `identity:get_group(group_id=bba303b549124f5898fa58d1e1f31db9)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:51.907129 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00107eff-3b29-4121-aeae-30c5203403f1 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:51.913294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 217/436] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 4 00:46:51 2022] GET /identity/v3/groups/bba303b549124f5898fa58d1e1f31db9 => generated 226 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:51.914178 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00107eff-3b29-4121-aeae-30c5203403f1 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:52.012089 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff46310-6010-44aa-b341-5cb4b5c22f42 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:52.048855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00107eff-3b29-4121-aeae-30c5203403f1 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:52.050631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:52.054229 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5aa3e8e-eabd-4a79-bbc3-8fc3bea7e6bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:52.054794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aa3e8e-eabd-4a79-bbc3-8fc3bea7e6bd None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:52.055073 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aa3e8e-eabd-4a79-bbc3-8fc3bea7e6bd None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:52.055342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5aa3e8e-eabd-4a79-bbc3-8fc3bea7e6bd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:52.092476 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49694ac-6af3-42ec-8db7-c4bde281dc8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:52.093272 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49694ac-6af3-42ec-8db7-c4bde281dc8c None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:52.093625 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49694ac-6af3-42ec-8db7-c4bde281dc8c None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:52.094074 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49694ac-6af3-42ec-8db7-c4bde281dc8c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:52.101165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:52.105030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49694ac-6af3-42ec-8db7-c4bde281dc8c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:52.110154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-f49694ac-6af3-42ec-8db7-c4bde281dc8c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:52.110154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:52.110154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:52.110154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:52.110154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:52.110154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:52.110154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:52.110154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:52.110154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:52.110154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:52.110154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:52.110154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:52.110154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:52.110154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:52.110154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:52.110154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:52.116695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:52.116695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:52.116695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:52.116695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:52.116695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:52.116695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:52.116695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:52.116695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:52.116695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:52.116695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:52.116695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:52.116695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:52.116695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:52.116695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:52.116695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:52.116695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:52.117738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:52.117738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:52.117738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:52.117738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:52.117738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:52.117738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:52.117738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:52.117738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:52.117738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:52.117738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:52.117738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:52.117738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:52.117738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 218/437] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:46:51 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:52.117738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49694ac-6af3-42ec-8db7-c4bde281dc8c None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:52.135484 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5aa3e8e-eabd-4a79-bbc3-8fc3bea7e6bd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:52.137100 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5aa3e8e-eabd-4a79-bbc3-8fc3bea7e6bd None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:52.222573 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49694ac-6af3-42ec-8db7-c4bde281dc8c None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:52.249777 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 220/438] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:52.250610 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5aa3e8e-eabd-4a79-bbc3-8fc3bea7e6bd None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:52.261667 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dc6120-b5fe-4de0-9259-313c4628e9a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:52.262339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc6120-b5fe-4de0-9259-313c4628e9a3 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:52.262620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc6120-b5fe-4de0-9259-313c4628e9a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:52.262913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc6120-b5fe-4de0-9259-313c4628e9a3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:52.263641 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dc6120-b5fe-4de0-9259-313c4628e9a3 None admin] RBAC: Adding query filter params (group.id=bba303b549124f5898fa58d1e1f31db9, role.id=c599b217579e42a49c657c642ea0648f, scope.project.id=1193eb16a76d432faf6141e47cc99c5a) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:52.263956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dc6120-b5fe-4de0-9259-313c4628e9a3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:52.265421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dc6120-b5fe-4de0-9259-313c4628e9a3 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:52.271903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 219/439] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 4 00:46:52 2022] GET /identity/v3/role_assignments?group.id=bba303b549124f5898fa58d1e1f31db9&scope.project.id=1193eb16a76d432faf6141e47cc99c5a&role.id=c599b217579e42a49c657c642ea0648f => generated 611 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:52.273036 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec237f2f-fced-4a75-8db7-4ac6d7af1637 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:52.273349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec237f2f-fced-4a75-8db7-4ac6d7af1637 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:52.273622 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec237f2f-fced-4a75-8db7-4ac6d7af1637 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:52.274818 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 220/440] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:52 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:52.276786 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec237f2f-fced-4a75-8db7-4ac6d7af1637 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:52.314450 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:52.335539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:52.367930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5aa3e8e-eabd-4a79-bbc3-8fc3bea7e6bd None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:52.398909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec237f2f-fced-4a75-8db7-4ac6d7af1637 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:52.406177 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ad5636-42f3-4aae-a039-e9a16c7b3b35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:52.406789 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ad5636-42f3-4aae-a039-e9a16c7b3b35 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:52.407072 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ad5636-42f3-4aae-a039-e9a16c7b3b35 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:52.407340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ad5636-42f3-4aae-a039-e9a16c7b3b35 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:52.408007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ad5636-42f3-4aae-a039-e9a16c7b3b35 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:52.408305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ad5636-42f3-4aae-a039-e9a16c7b3b35 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:52.409917 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ad5636-42f3-4aae-a039-e9a16c7b3b35 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:52.418423 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 221/441] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:46:52 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:52.420853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25589a88-4660-4801-9263-17514fea2652 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:52.420853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25589a88-4660-4801-9263-17514fea2652 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:52.420853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25589a88-4660-4801-9263-17514fea2652 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:52.443401 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-25589a88-4660-4801-9263-17514fea2652 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:52.444597 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-25589a88-4660-4801-9263-17514fea2652 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:52.454401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e45a77a-38a1-439b-ac05-475ee64e5787 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:52.455306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e45a77a-38a1-439b-ac05-475ee64e5787 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:52.455755 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e45a77a-38a1-439b-ac05-475ee64e5787 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:52.456298 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e45a77a-38a1-439b-ac05-475ee64e5787 None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:52.460384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e45a77a-38a1-439b-ac05-475ee64e5787 None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:52.461870 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e45a77a-38a1-439b-ac05-475ee64e5787 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:52.463869 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 221/442] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:52 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:52.464789 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e45a77a-38a1-439b-ac05-475ee64e5787 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:52.528917 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 222/443] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:52.530224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25589a88-4660-4801-9263-17514fea2652 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:52.594128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:52.600636 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e45a77a-38a1-439b-ac05-475ee64e5787 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:52.619231 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:52.647590 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f439ec-65df-4c1f-bf3f-e0a5b9b1611d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:52.648211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f439ec-65df-4c1f-bf3f-e0a5b9b1611d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:52.648485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f439ec-65df-4c1f-bf3f-e0a5b9b1611d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:52.648984 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f439ec-65df-4c1f-bf3f-e0a5b9b1611d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:52.675281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25589a88-4660-4801-9263-17514fea2652 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:52.716441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2987566f-d093-434f-9361-82ae3de99282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:52.717302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2987566f-d093-434f-9361-82ae3de99282 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:52.717480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2987566f-d093-434f-9361-82ae3de99282 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:52.717754 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2987566f-d093-434f-9361-82ae3de99282 None admin] PATH_INFO: `/v3/groups/bba303b549124f5898fa58d1e1f31db9` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:52.718866 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2987566f-d093-434f-9361-82ae3de99282 None admin] RBAC: Authorizing `identity:get_group(group_id=bba303b549124f5898fa58d1e1f31db9)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:52.720304 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2987566f-d093-434f-9361-82ae3de99282 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:52.722079 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 223/444] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 4 00:46:52 2022] GET /identity/v3/groups/bba303b549124f5898fa58d1e1f31db9 => generated 226 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:52.728261 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2987566f-d093-434f-9361-82ae3de99282 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:52.730873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f439ec-65df-4c1f-bf3f-e0a5b9b1611d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:52.732308 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f439ec-65df-4c1f-bf3f-e0a5b9b1611d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:52.825797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:52.830579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2987566f-d093-434f-9361-82ae3de99282 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:52.834097 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 222/445] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:46:52.834817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f439ec-65df-4c1f-bf3f-e0a5b9b1611d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:52.861492 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:52.865512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70c462c-d298-49bd-86e6-b9d6942a9d6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:52.866111 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70c462c-d298-49bd-86e6-b9d6942a9d6c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:52.866385 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70c462c-d298-49bd-86e6-b9d6942a9d6c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:52.866656 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70c462c-d298-49bd-86e6-b9d6942a9d6c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:52.867368 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70c462c-d298-49bd-86e6-b9d6942a9d6c None admin] RBAC: Adding query filter params (group.id=bba303b549124f5898fa58d1e1f31db9, role.id=4bfd6d0544ee4a9e933538e78135a12c, scope.project.id=d737f411ca514fb4aa93ee11d55926ae) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:52.867671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70c462c-d298-49bd-86e6-b9d6942a9d6c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:52.869098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70c462c-d298-49bd-86e6-b9d6942a9d6c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:52.874979 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 224/446] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 4 00:46:52 2022] GET /identity/v3/role_assignments?group.id=bba303b549124f5898fa58d1e1f31db9&scope.project.id=d737f411ca514fb4aa93ee11d55926ae&role.id=4bfd6d0544ee4a9e933538e78135a12c => generated 611 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:52.875743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70c462c-d298-49bd-86e6-b9d6942a9d6c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:52.939452 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f439ec-65df-4c1f-bf3f-e0a5b9b1611d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:52.985390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa34d2d7-ef01-4bc3-b0fd-adfdb6b5f12f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:52.986046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa34d2d7-ef01-4bc3-b0fd-adfdb6b5f12f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:52.986323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa34d2d7-ef01-4bc3-b0fd-adfdb6b5f12f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:52.986594 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa34d2d7-ef01-4bc3-b0fd-adfdb6b5f12f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:52.992784 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70c462c-d298-49bd-86e6-b9d6942a9d6c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:52.999438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa34d2d7-ef01-4bc3-b0fd-adfdb6b5f12f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:53.004431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-aa34d2d7-ef01-4bc3-b0fd-adfdb6b5f12f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:53.004431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:53.004431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:53.004431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:53.004431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:53.004431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:53.004431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:53.004431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:53.004431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:53.004431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:53.004431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:53.004431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:53.004431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:53.004431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:53.004431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:53.004431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:53.005466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:53.005466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:53.005466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:53.005466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:53.005466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:53.005466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:53.005466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:53.005466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:53.005466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:53.005466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:53.005466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:53.005466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:53.005466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:53.005466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:53.005466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:53.005466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:53.006248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:53.006248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:53.006248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:53.006248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:53.006248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:53.006248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:53.006248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:53.006248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:53.006248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:53.006248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:53.006248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:46:53.006248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:53.006248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 223/447] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:46:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:53.011593 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa34d2d7-ef01-4bc3-b0fd-adfdb6b5f12f None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:53.038031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efca0e4d-eb61-444a-9876-82a0f34db6e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:53.039124 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efca0e4d-eb61-444a-9876-82a0f34db6e7 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:53.039531 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efca0e4d-eb61-444a-9876-82a0f34db6e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:53.039863 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efca0e4d-eb61-444a-9876-82a0f34db6e7 None admin] PATH_INFO: `/v3/projects/d737f411ca514fb4aa93ee11d55926ae` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:53.041389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efca0e4d-eb61-444a-9876-82a0f34db6e7 None admin] RBAC: Authorizing `identity:get_project(project_id=d737f411ca514fb4aa93ee11d55926ae)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:53.042960 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efca0e4d-eb61-444a-9876-82a0f34db6e7 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:53.045273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 225/448] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:52 2022] GET /identity/v3/projects/d737f411ca514fb4aa93ee11d55926ae => generated 302 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:53.051762 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efca0e4d-eb61-444a-9876-82a0f34db6e7 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:53.077254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:53.128726 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:53.130569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa34d2d7-ef01-4bc3-b0fd-adfdb6b5f12f None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:53.175508 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2452fff-b654-4052-bd82-26c774c8fec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:53.176595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2452fff-b654-4052-bd82-26c774c8fec3 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:53.176962 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2452fff-b654-4052-bd82-26c774c8fec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:53.179909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2452fff-b654-4052-bd82-26c774c8fec3 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:53.180024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efca0e4d-eb61-444a-9876-82a0f34db6e7 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:53.180103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2452fff-b654-4052-bd82-26c774c8fec3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:53.180103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2452fff-b654-4052-bd82-26c774c8fec3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:53.180103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2452fff-b654-4052-bd82-26c774c8fec3 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:53.192025 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 224/449] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:46:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:53.195463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2452fff-b654-4052-bd82-26c774c8fec3 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:53.217339 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838f64a1-c759-41b1-ac8c-60872a0c6b21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:53.218000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838f64a1-c759-41b1-ac8c-60872a0c6b21 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:53.218282 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838f64a1-c759-41b1-ac8c-60872a0c6b21 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:53.218545 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838f64a1-c759-41b1-ac8c-60872a0c6b21 None admin] PATH_INFO: `/v3/groups/bba303b549124f5898fa58d1e1f31db9` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:53.219747 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838f64a1-c759-41b1-ac8c-60872a0c6b21 None admin] RBAC: Authorizing `identity:get_group(group_id=bba303b549124f5898fa58d1e1f31db9)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:53.221233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838f64a1-c759-41b1-ac8c-60872a0c6b21 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:53.223214 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 226/450] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 4 00:46:53 2022] GET /identity/v3/groups/bba303b549124f5898fa58d1e1f31db9 => generated 226 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:53.228743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838f64a1-c759-41b1-ac8c-60872a0c6b21 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:53.312318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2452fff-b654-4052-bd82-26c774c8fec3 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:53.328830 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:53.333651 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838f64a1-c759-41b1-ac8c-60872a0c6b21 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:53.346028 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d3a774-f16a-4af3-9602-6048c135e1b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:53.346636 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d3a774-f16a-4af3-9602-6048c135e1b0 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:53.346921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d3a774-f16a-4af3-9602-6048c135e1b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:53.347196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d3a774-f16a-4af3-9602-6048c135e1b0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:53.362162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:53.368754 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9670afaa-c833-4c60-8a0b-8a96f13b5e4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:53.369380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9670afaa-c833-4c60-8a0b-8a96f13b5e4d None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:53.370680 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9670afaa-c833-4c60-8a0b-8a96f13b5e4d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:53.370680 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9670afaa-c833-4c60-8a0b-8a96f13b5e4d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:53.370680 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9670afaa-c833-4c60-8a0b-8a96f13b5e4d None admin] RBAC: Adding query filter params (group.id=bba303b549124f5898fa58d1e1f31db9, role.id=c599b217579e42a49c657c642ea0648f, scope.project.id=d737f411ca514fb4aa93ee11d55926ae) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:53.371051 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9670afaa-c833-4c60-8a0b-8a96f13b5e4d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:53.372488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9670afaa-c833-4c60-8a0b-8a96f13b5e4d None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:53.381166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 227/451] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 4 00:46:53 2022] GET /identity/v3/role_assignments?group.id=bba303b549124f5898fa58d1e1f31db9&scope.project.id=d737f411ca514fb4aa93ee11d55926ae&role.id=c599b217579e42a49c657c642ea0648f => generated 611 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:46:53.424411 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d3a774-f16a-4af3-9602-6048c135e1b0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:53.426487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d3a774-f16a-4af3-9602-6048c135e1b0 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:53.533857 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 225/452] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:46:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 04 00:46:53.540420 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9670afaa-c833-4c60-8a0b-8a96f13b5e4d None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:53.642259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9670afaa-c833-4c60-8a0b-8a96f13b5e4d None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:53.676120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd9ea98-2b49-424e-807b-8711d141bc2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:53.676746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd9ea98-2b49-424e-807b-8711d141bc2f None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:53.677029 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd9ea98-2b49-424e-807b-8711d141bc2f None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:53.677411 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd9ea98-2b49-424e-807b-8711d141bc2f None admin] PATH_INFO: `/v3/projects/f25649add4b1426590abafbe586fc1fa` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:53.678547 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd9ea98-2b49-424e-807b-8711d141bc2f None admin] RBAC: Authorizing `identity:get_project(project_id=f25649add4b1426590abafbe586fc1fa)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:53.679981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd9ea98-2b49-424e-807b-8711d141bc2f None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:53.681640 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 228/453] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:46:53 2022] GET /identity/v3/projects/f25649add4b1426590abafbe586fc1fa => generated 344 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:53.687436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d3a774-f16a-4af3-9602-6048c135e1b0 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:53.705684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:53.784013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d3a774-f16a-4af3-9602-6048c135e1b0 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:53.816696 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1d865a-c391-49ec-9dbb-5277c639e838 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:53.817333 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1d865a-c391-49ec-9dbb-5277c639e838 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:53.817606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1d865a-c391-49ec-9dbb-5277c639e838 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:53.817874 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1d865a-c391-49ec-9dbb-5277c639e838 None admin] PATH_INFO: `/v3/groups/d548703b5e7a4d17be559d2a5f32aa9c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:53.818952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1d865a-c391-49ec-9dbb-5277c639e838 None admin] RBAC: Authorizing `identity:get_group(group_id=d548703b5e7a4d17be559d2a5f32aa9c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:53.820372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1d865a-c391-49ec-9dbb-5277c639e838 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:53.821930 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 226/454] 10.222.0.42 () {56 vars in 1180 bytes} [Sun Sep 4 00:46:53 2022] GET /identity/v3/groups/d548703b5e7a4d17be559d2a5f32aa9c => generated 229 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:53.827774 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd9ea98-2b49-424e-807b-8711d141bc2f None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:53.887930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:53.925130 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd9ea98-2b49-424e-807b-8711d141bc2f None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:53.958059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4a5edf-9ce9-495e-9b2f-ddde54213aa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:53.958646 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4a5edf-9ce9-495e-9b2f-ddde54213aa0 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:53.958921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4a5edf-9ce9-495e-9b2f-ddde54213aa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:53.959191 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4a5edf-9ce9-495e-9b2f-ddde54213aa0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:53.959905 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4a5edf-9ce9-495e-9b2f-ddde54213aa0 None admin] RBAC: Adding query filter params (group.id=d548703b5e7a4d17be559d2a5f32aa9c, role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=f25649add4b1426590abafbe586fc1fa) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:53.960209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4a5edf-9ce9-495e-9b2f-ddde54213aa0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:53.961665 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4a5edf-9ce9-495e-9b2f-ddde54213aa0 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:53.973688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 229/455] 10.222.0.42 () {56 vars in 1376 bytes} [Sun Sep 4 00:46:53 2022] GET /identity/v3/role_assignments?group.id=d548703b5e7a4d17be559d2a5f32aa9c&scope.project.id=f25649add4b1426590abafbe586fc1fa&role.id=19c006412bec45dc8febd7cc86b9678f => generated 611 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:56.525215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32bb583-7e43-49c8-b53e-9093bbf4d40d None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:56.525729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32bb583-7e43-49c8-b53e-9093bbf4d40d None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:56.525729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32bb583-7e43-49c8-b53e-9093bbf4d40d None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:56.527298 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 227/456] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:56.532824 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4183bff-d88d-4dbf-8d67-2efdc49fad3e None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:56.533258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4183bff-d88d-4dbf-8d67-2efdc49fad3e None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:56.537363 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4183bff-d88d-4dbf-8d67-2efdc49fad3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:56.562658 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-a4183bff-d88d-4dbf-8d67-2efdc49fad3e None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:56.563852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a4183bff-d88d-4dbf-8d67-2efdc49fad3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:56.648679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 230/457] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:56.682117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4731eb44-bdea-4df4-b879-87dd0e616ec1 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:56.682429 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4731eb44-bdea-4df4-b879-87dd0e616ec1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:56.682711 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4731eb44-bdea-4df4-b879-87dd0e616ec1 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:56.684007 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 228/458] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:56.689736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55877089-7607-46eb-9fba-776e343bbc67 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:56.690052 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55877089-7607-46eb-9fba-776e343bbc67 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:56.690286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55877089-7607-46eb-9fba-776e343bbc67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:56.711971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-55877089-7607-46eb-9fba-776e343bbc67 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:56.713103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-55877089-7607-46eb-9fba-776e343bbc67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:56.781752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:56.802703 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 231/459] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:57.078107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24301fe6-287b-433c-b2c1-8e07b880c9e8 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:57.078782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24301fe6-287b-433c-b2c1-8e07b880c9e8 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:57.079284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24301fe6-287b-433c-b2c1-8e07b880c9e8 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:57.081148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 229/460] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:57.082370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55877089-7607-46eb-9fba-776e343bbc67 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:57.082900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ed93bb-8d83-4902-a41f-46d3a92f5a7d None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:57.083185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ed93bb-8d83-4902-a41f-46d3a92f5a7d None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:57.083457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ed93bb-8d83-4902-a41f-46d3a92f5a7d None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:57.084695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 230/461] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:57.089836 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f697c0-dcb7-4ff7-80bc-7080dfb7cf92 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:57.090126 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f697c0-dcb7-4ff7-80bc-7080dfb7cf92 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:57.090399 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f697c0-dcb7-4ff7-80bc-7080dfb7cf92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:57.112070 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-e6f697c0-dcb7-4ff7-80bc-7080dfb7cf92 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:57.113853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e6f697c0-dcb7-4ff7-80bc-7080dfb7cf92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:57.184132 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 231/462] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:46:57.190216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06d07bd-72fb-4291-a0e7-0984550131f6 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:57.190503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06d07bd-72fb-4291-a0e7-0984550131f6 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:57.190784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06d07bd-72fb-4291-a0e7-0984550131f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:57.195172 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55877089-7607-46eb-9fba-776e343bbc67 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:57.217815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-e06d07bd-72fb-4291-a0e7-0984550131f6 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:57.219011 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e06d07bd-72fb-4291-a0e7-0984550131f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:57.229324 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:57.239650 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6298c170-f027-4233-a163-9b81b1d16a17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:57.240288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6298c170-f027-4233-a163-9b81b1d16a17 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:57.240567 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6298c170-f027-4233-a163-9b81b1d16a17 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:57.240838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6298c170-f027-4233-a163-9b81b1d16a17 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:57.251719 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6298c170-f027-4233-a163-9b81b1d16a17 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:57.254140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6298c170-f027-4233-a163-9b81b1d16a17 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:57.269727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-6298c170-f027-4233-a163-9b81b1d16a17 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:46:57.269727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:57.269727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:57.269727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:57.269727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:57.269727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:57.269727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:57.269727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:57.269727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:57.269727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:57.269727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:57.269727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:57.269727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:46:57.269727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:46:57.269727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:46:57.269727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:46:57.270660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:57.270660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:57.270660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:57.270660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:57.270660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:57.270660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:57.270660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:57.270660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:57.270660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:57.270660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:57.270660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:57.270660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:57.270660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:57.270660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:57.270660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:57.270660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:57.270660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:46:57.271733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:46:57.271733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:46:57.271733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:57.271733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 232/463] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:46:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:57.271968 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6298c170-f027-4233-a163-9b81b1d16a17 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:57.289761 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 232/464] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:57.293912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06d07bd-72fb-4291-a0e7-0984550131f6 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:57.320931 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:57.387861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6298c170-f027-4233-a163-9b81b1d16a17 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:57.425255 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dd6445-f666-410b-9d49-0f4364565e35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:57.426981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dd6445-f666-410b-9d49-0f4364565e35 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:57.427440 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dd6445-f666-410b-9d49-0f4364565e35 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:57.427885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dd6445-f666-410b-9d49-0f4364565e35 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:57.438937 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dd6445-f666-410b-9d49-0f4364565e35 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:57.440655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dd6445-f666-410b-9d49-0f4364565e35 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:57.449816 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06d07bd-72fb-4291-a0e7-0984550131f6 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:57.457947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-05dd6445-f666-410b-9d49-0f4364565e35 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:46:57.457947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:57.457947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:57.457947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:57.457947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:57.457947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:57.457947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:57.457947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:57.457947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:57.457947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:57.457947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:57.457947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:57.457947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:46:57.457947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:46:57.457947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:46:57.457947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:46:57.458980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:57.458980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:57.458980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:57.458980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:57.458980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:57.458980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:57.458980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:57.458980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:57.458980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:57.458980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:57.458980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:57.458980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:57.458980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:57.458980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:57.458980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:57.458980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:57.458980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:46:57.459810 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:46:57.459810 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:46:57.459810 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:57.461580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 233/465] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:46:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:57.470103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dd6445-f666-410b-9d49-0f4364565e35 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:57.502793 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d21e64-f5ed-4a45-a11e-84cf9dc7e95c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:57.504016 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d21e64-f5ed-4a45-a11e-84cf9dc7e95c None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:57.504448 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d21e64-f5ed-4a45-a11e-84cf9dc7e95c None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:57.504815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d21e64-f5ed-4a45-a11e-84cf9dc7e95c None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:57.505506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d21e64-f5ed-4a45-a11e-84cf9dc7e95c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:57.505817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d21e64-f5ed-4a45-a11e-84cf9dc7e95c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:57.507191 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d21e64-f5ed-4a45-a11e-84cf9dc7e95c None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:57.524528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 233/466] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:46:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:57.525906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d21e64-f5ed-4a45-a11e-84cf9dc7e95c None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:57.531762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:57.608393 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:57.628969 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dd6445-f666-410b-9d49-0f4364565e35 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:57.647874 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d21e64-f5ed-4a45-a11e-84cf9dc7e95c None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:57.672912 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92be274-8c1e-4f66-bcb2-0b099d05dd77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:57.673861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92be274-8c1e-4f66-bcb2-0b099d05dd77 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:57.674409 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92be274-8c1e-4f66-bcb2-0b099d05dd77 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:57.674855 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92be274-8c1e-4f66-bcb2-0b099d05dd77 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:57.676000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92be274-8c1e-4f66-bcb2-0b099d05dd77 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:57.676710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92be274-8c1e-4f66-bcb2-0b099d05dd77 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:57.678700 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92be274-8c1e-4f66-bcb2-0b099d05dd77 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:57.683832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfbdf1e-85a6-448d-8d16-38c431503ef0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:57.684504 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfbdf1e-85a6-448d-8d16-38c431503ef0 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:57.684882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfbdf1e-85a6-448d-8d16-38c431503ef0 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:57.685344 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfbdf1e-85a6-448d-8d16-38c431503ef0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:57.698390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfbdf1e-85a6-448d-8d16-38c431503ef0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:57.699967 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfbdf1e-85a6-448d-8d16-38c431503ef0 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:57.701004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 234/467] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:46:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:57.701852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92be274-8c1e-4f66-bcb2-0b099d05dd77 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:57.717506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-fbfbdf1e-85a6-448d-8d16-38c431503ef0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:46:57.717506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:57.717506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:57.717506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:57.717506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:57.717506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:57.717506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:57.717506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:57.717506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:57.717506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:57.717506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:57.717506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:57.717506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:46:57.717506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:46:57.717506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:46:57.717506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:46:57.718455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:57.718455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:57.718455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:57.718455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:57.718455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:57.718455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:57.718455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:57.718455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:57.718455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:57.718455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:57.718455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:57.718455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:57.718455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:57.718455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:57.718455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:57.718455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:57.718455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:46:57.719295 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:46:57.719295 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:46:57.719295 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:57.719295 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 234/468] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:46:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:57.719526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfbdf1e-85a6-448d-8d16-38c431503ef0 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:57.785025 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:57.808130 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92be274-8c1e-4f66-bcb2-0b099d05dd77 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:57.831052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfbdf1e-85a6-448d-8d16-38c431503ef0 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:57.845415 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e70a41-8876-4b54-a83d-0515517a2bbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:57.846026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e70a41-8876-4b54-a83d-0515517a2bbd None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:57.846306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e70a41-8876-4b54-a83d-0515517a2bbd None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:57.846579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e70a41-8876-4b54-a83d-0515517a2bbd None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:57.847256 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e70a41-8876-4b54-a83d-0515517a2bbd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:57.848696 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e70a41-8876-4b54-a83d-0515517a2bbd None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:57.866067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2adaad-7a8e-4a2a-845c-869a78b8114e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:57.866688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2adaad-7a8e-4a2a-845c-869a78b8114e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:57.866964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2adaad-7a8e-4a2a-845c-869a78b8114e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:57.867235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2adaad-7a8e-4a2a-845c-869a78b8114e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:57.877931 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2adaad-7a8e-4a2a-845c-869a78b8114e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:57.879989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2adaad-7a8e-4a2a-845c-869a78b8114e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:57.893823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-4e2adaad-7a8e-4a2a-845c-869a78b8114e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:46:57.893823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:57.893823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:57.893823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:57.893823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:57.893823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:57.893823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:57.893823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:57.893823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:57.893823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:57.893823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:57.893823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:57.893823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:46:57.893823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:46:57.893823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:46:57.893823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:46:57.895716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:57.895716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:57.895716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:57.895716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:57.895716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:57.895716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:57.895716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:57.895716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:57.895716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:57.895716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:57.895716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:57.895716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:57.895716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:57.895716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:57.895716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:57.895716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:57.895716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:46:57.896825 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:46:57.896825 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:46:57.896825 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:57.896825 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 235/469] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:46:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:57.896825 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2adaad-7a8e-4a2a-845c-869a78b8114e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:57.935712 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 235/470] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:46:57 2022] POST /identity/v3/users => generated 248 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:46:57.936634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e70a41-8876-4b54-a83d-0515517a2bbd None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:57.947319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:58.023347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2adaad-7a8e-4a2a-845c-869a78b8114e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:58.033658 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:58.061429 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e70a41-8876-4b54-a83d-0515517a2bbd None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:58.064859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c3be53-017a-4f14-9e57-98beaa0a76e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:58.065529 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c3be53-017a-4f14-9e57-98beaa0a76e0 None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:58.066451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c3be53-017a-4f14-9e57-98beaa0a76e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:58.066895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c3be53-017a-4f14-9e57-98beaa0a76e0 None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:58.067523 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c3be53-017a-4f14-9e57-98beaa0a76e0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:58.069026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c3be53-017a-4f14-9e57-98beaa0a76e0 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:58.097859 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5f8fcf-548e-43f0-a6de-74695a0db3dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:58.098756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5f8fcf-548e-43f0-a6de-74695a0db3dd None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:58.099150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5f8fcf-548e-43f0-a6de-74695a0db3dd None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:58.099432 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5f8fcf-548e-43f0-a6de-74695a0db3dd None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:58.100085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5f8fcf-548e-43f0-a6de-74695a0db3dd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:58.100392 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5f8fcf-548e-43f0-a6de-74695a0db3dd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:58.101897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5f8fcf-548e-43f0-a6de-74695a0db3dd None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:58.115246 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 236/471] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:46:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:58.116237 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2c2f81-af95-48f0-8d8e-a928f8d073d0 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:58.116348 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2c2f81-af95-48f0-8d8e-a928f8d073d0 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:58.116545 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2c2f81-af95-48f0-8d8e-a928f8d073d0 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:58.117764 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 237/472] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:46:58 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:46:58.118564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2c2f81-af95-48f0-8d8e-a928f8d073d0 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:58.182737 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 236/473] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:46:57 2022] POST /identity/v3/users => generated 246 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:46:58.190532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c3be53-017a-4f14-9e57-98beaa0a76e0 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:58.191717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:58.243840 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2c2f81-af95-48f0-8d8e-a928f8d073d0 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:58.295308 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40609ea7-4eb9-4c85-8d2d-d63cb462d9a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:58.295935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40609ea7-4eb9-4c85-8d2d-d63cb462d9a1 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:58.296213 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40609ea7-4eb9-4c85-8d2d-d63cb462d9a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:58.296480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40609ea7-4eb9-4c85-8d2d-d63cb462d9a1 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:58.297177 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40609ea7-4eb9-4c85-8d2d-d63cb462d9a1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:58.297484 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40609ea7-4eb9-4c85-8d2d-d63cb462d9a1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:58.298965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40609ea7-4eb9-4c85-8d2d-d63cb462d9a1 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:58.312441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 238/474] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:46:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:46:58.313634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e337fea4-405f-4693-8e76-a8acc5fa4a02 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:58.313975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e337fea4-405f-4693-8e76-a8acc5fa4a02 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:58.314251 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e337fea4-405f-4693-8e76-a8acc5fa4a02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:58.334407 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-e337fea4-405f-4693-8e76-a8acc5fa4a02 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:46:58.335613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e337fea4-405f-4693-8e76-a8acc5fa4a02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:46:58.338260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c3be53-017a-4f14-9e57-98beaa0a76e0 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:58.381352 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196fa5ca-01d3-464e-8b75-03cdee72210b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:58.382289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196fa5ca-01d3-464e-8b75-03cdee72210b None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:58.382804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196fa5ca-01d3-464e-8b75-03cdee72210b None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:58.383253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196fa5ca-01d3-464e-8b75-03cdee72210b None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:58.384098 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196fa5ca-01d3-464e-8b75-03cdee72210b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:58.385783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196fa5ca-01d3-464e-8b75-03cdee72210b None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:58.421017 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 239/475] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:46:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:46:58.421899 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e337fea4-405f-4693-8e76-a8acc5fa4a02 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:58.484649 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:58.501166 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 237/476] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:46:58 2022] POST /identity/v3/users => generated 249 bytes in 305 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:46:58.542088 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e337fea4-405f-4693-8e76-a8acc5fa4a02 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:58.585260 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99eb89c-33dd-4148-a7c7-50d5e32cdc39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:58.585950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99eb89c-33dd-4148-a7c7-50d5e32cdc39 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:58.586232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99eb89c-33dd-4148-a7c7-50d5e32cdc39 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:58.586507 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99eb89c-33dd-4148-a7c7-50d5e32cdc39 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:58.587228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99eb89c-33dd-4148-a7c7-50d5e32cdc39 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:58.588820 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99eb89c-33dd-4148-a7c7-50d5e32cdc39 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:58.673930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 240/477] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:46:58 2022] POST /identity/v3/users => generated 248 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:46:58.705150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196fa5ca-01d3-464e-8b75-03cdee72210b None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:58.735306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:58.827512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196fa5ca-01d3-464e-8b75-03cdee72210b None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:58.868438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d296c5-c737-415a-81f1-9b5221a86850 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:58.869057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d296c5-c737-415a-81f1-9b5221a86850 None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:58.869437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d296c5-c737-415a-81f1-9b5221a86850 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:58.869722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d296c5-c737-415a-81f1-9b5221a86850 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:58.870484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d296c5-c737-415a-81f1-9b5221a86850 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:58.871918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d296c5-c737-415a-81f1-9b5221a86850 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:58.913633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.sql.core [None req-14d296c5-c737-415a-81f1-9b5221a86850 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 04 00:46:58.913633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 04 00:46:58.913633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [parameters: {'id': '5b33d64b56f24e83b8c8ce099c82ec87', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 04 00:46:58.913633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81559) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 04 00:46:58.931936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-14d296c5-c737-415a-81f1-9b5221a86850 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 04 00:46:58.931936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:58.931936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 04 00:46:58.931936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 04 00:46:58.931936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 04 00:46:58.931936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 04 00:46:58.931936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 04 00:46:58.931936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application result = self._query(query) Sep 04 00:46:58.931936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 04 00:46:58.931936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application conn.query(q) Sep 04 00:46:58.931936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 04 00:46:58.931936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 04 00:46:58.931936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 04 00:46:58.931936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application result.read() Sep 04 00:46:58.931936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return ref.to_dict() Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 04 00:46:58.932833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application next(self.gen) Sep 04 00:46:58.934100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 04 00:46:58.934100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application yield resource Sep 04 00:46:58.934100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 04 00:46:58.934100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application next(self.gen) Sep 04 00:46:58.934100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 04 00:46:58.934100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self.session.rollback() Sep 04 00:46:58.934100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 04 00:46:58.934100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self.force_reraise() Sep 04 00:46:58.934100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 04 00:46:58.934100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise self.value Sep 04 00:46:58.934100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 04 00:46:58.934100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 04 00:46:58.934100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 04 00:46:58.934100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application session.commit() Sep 04 00:46:58.934100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 04 00:46:58.934100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 04 00:46:58.934100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 04 00:46:58.935188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._prepare_impl() Sep 04 00:46:58.935188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 04 00:46:58.935188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self.session.flush() Sep 04 00:46:58.935188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 04 00:46:58.935188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._flush(objects) Sep 04 00:46:58.935188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 04 00:46:58.935188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 04 00:46:58.935188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 04 00:46:58.935188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application compat.raise_( Sep 04 00:46:58.935188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 04 00:46:58.935188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception Sep 04 00:46:58.935188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 04 00:46:58.935188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application flush_context.execute() Sep 04 00:46:58.935188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 04 00:46:58.935188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rec.execute(self) Sep 04 00:46:58.935188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 04 00:46:58.935188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 04 00:46:58.936034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 04 00:46:58.936034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 04 00:46:58.936034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 04 00:46:58.936034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 04 00:46:58.936034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 04 00:46:58.936034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 04 00:46:58.936034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 04 00:46:58.936034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 04 00:46:58.936034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 04 00:46:58.936034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 04 00:46:58.936034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 04 00:46:58.936034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 04 00:46:58.936034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 04 00:46:58.936034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 04 00:46:58.936034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 04 00:46:58.936034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception Sep 04 00:46:58.936804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 04 00:46:58.936804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 04 00:46:58.936804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 04 00:46:58.936804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 04 00:46:58.936804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 04 00:46:58.936804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application result = self._query(query) Sep 04 00:46:58.936804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 04 00:46:58.936804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application conn.query(q) Sep 04 00:46:58.936804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 04 00:46:58.936804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 04 00:46:58.936804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 04 00:46:58.936804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application result.read() Sep 04 00:46:58.936804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 04 00:46:58.936804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 04 00:46:58.936804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 04 00:46:58.936804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 04 00:46:58.937621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 04 00:46:58.937621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 04 00:46:58.937621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 04 00:46:58.937621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 04 00:46:58.937621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 04 00:46:58.937621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 04 00:46:58.937621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [parameters: {'id': '5b33d64b56f24e83b8c8ce099c82ec87', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 04 00:46:58.937621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 04 00:46:58.937621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:58.937621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:46:58.937621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:58.937621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:58.937621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:58.937621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:58.937621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:58.937621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:58.938509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:58.938509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:58.938509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:58.938509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:58.938509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:58.938509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:58.938509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 04 00:46:58.938509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 04 00:46:58.938509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:58.938509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:58.938509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 04 00:46:58.938509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 04 00:46:58.938509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 04 00:46:58.938509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 04 00:46:58.938509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 04 00:46:58.938509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:46:58.939288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 238/478] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Sep 4 00:46:58 2022] POST /identity/v3/roles => generated 144 bytes in 229 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:58.943917 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99eb89c-33dd-4148-a7c7-50d5e32cdc39 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:59.088053 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99eb89c-33dd-4148-a7c7-50d5e32cdc39 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:59.104589 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:59.136396 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2305ea8e-fa52-4f0f-8483-aca92bbe759c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:59.137102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2305ea8e-fa52-4f0f-8483-aca92bbe759c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:59.137396 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2305ea8e-fa52-4f0f-8483-aca92bbe759c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:59.137671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2305ea8e-fa52-4f0f-8483-aca92bbe759c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:59.155819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2305ea8e-fa52-4f0f-8483-aca92bbe759c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:46:59.161287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-2305ea8e-fa52-4f0f-8483-aca92bbe759c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:59.161287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:46:59.161287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:46:59.161287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:46:59.161287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:46:59.161287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:46:59.161287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:46:59.161287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:46:59.161287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:46:59.161287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:46:59.161287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:46:59.161287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:46:59.161287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:46:59.161287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:46:59.161287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:46:59.161287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:46:59.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:46:59.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:46:59.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:46:59.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:46:59.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:46:59.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:46:59.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:46:59.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:46:59.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:46:59.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:46:59.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:46:59.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:46:59.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:46:59.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:46:59.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:46:59.162143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:46:59.163327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:46:59.163327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:46:59.163327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:46:59.163327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:46:59.163327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:46:59.163327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:46:59.163327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:46:59.163327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:46:59.163327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:46:59.163327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:46:59.163327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 04 00:46:59.163327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:46:59.163327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 241/479] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:46:58 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:46:59.168283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d296c5-c737-415a-81f1-9b5221a86850 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:46:59.305744 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:46:59.325696 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d296c5-c737-415a-81f1-9b5221a86850 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:46:59.380371 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcf52dc-11af-48b9-8020-f3a1036bee1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:46:59.380913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcf52dc-11af-48b9-8020-f3a1036bee1d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:46:59.381222 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcf52dc-11af-48b9-8020-f3a1036bee1d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:46:59.381499 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcf52dc-11af-48b9-8020-f3a1036bee1d None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:46:59.382185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcf52dc-11af-48b9-8020-f3a1036bee1d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:46:59.382492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcf52dc-11af-48b9-8020-f3a1036bee1d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:46:59.383863 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fcf52dc-11af-48b9-8020-f3a1036bee1d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:46:59.396644 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 239/480] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:46:59 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:00.218746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599bedd5-62eb-477a-9159-3c6bfa04968d None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:00.219464 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599bedd5-62eb-477a-9159-3c6bfa04968d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:00.219464 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599bedd5-62eb-477a-9159-3c6bfa04968d None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:00.221032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 242/481] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:00.239194 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2641e69-2448-4b51-b9a1-eccdae5d99d6 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:00.239898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2641e69-2448-4b51-b9a1-eccdae5d99d6 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:00.239898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2641e69-2448-4b51-b9a1-eccdae5d99d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:00.262081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-f2641e69-2448-4b51-b9a1-eccdae5d99d6 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:00.263295 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f2641e69-2448-4b51-b9a1-eccdae5d99d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:00.361699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 240/482] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:00.608043 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca08d6a-e0d0-446c-b290-b7b8ab76850c None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:00.608373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca08d6a-e0d0-446c-b290-b7b8ab76850c None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:00.608588 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca08d6a-e0d0-446c-b290-b7b8ab76850c None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:00.610222 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 243/483] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:00.615217 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdca088-b044-4829-a425-12df41ad5c5b None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:00.615515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdca088-b044-4829-a425-12df41ad5c5b None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:00.615791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdca088-b044-4829-a425-12df41ad5c5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:00.639169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-0fdca088-b044-4829-a425-12df41ad5c5b None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:00.640343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-0fdca088-b044-4829-a425-12df41ad5c5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:00.725668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 241/484] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:00.810376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca08d6a-e0d0-446c-b290-b7b8ab76850c None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:00.947610 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca08d6a-e0d0-446c-b290-b7b8ab76850c None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:00.991517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5c90ad-b75b-40b4-956d-9e59667568a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:00.992524 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5c90ad-b75b-40b4-956d-9e59667568a9 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:00.993003 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5c90ad-b75b-40b4-956d-9e59667568a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:00.993512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5c90ad-b75b-40b4-956d-9e59667568a9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:01.010509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5c90ad-b75b-40b4-956d-9e59667568a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:01.010670 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90304c8-ee16-45fa-9344-47610c4f5625 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:01.010911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90304c8-ee16-45fa-9344-47610c4f5625 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:01.011197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90304c8-ee16-45fa-9344-47610c4f5625 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:01.012799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 242/485] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:01.017060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-3f5c90ad-b75b-40b4-956d-9e59667568a9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:01.017060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:01.017060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:01.017060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:01.017060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:01.017060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:01.017060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:01.017060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:01.017060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:01.017060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:01.017060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:01.017060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:01.017060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:01.017060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:01.017060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:01.017060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:01.017971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:01.017971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:01.017971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:01.017971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:01.017971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:01.017971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:01.017971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:01.017971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:01.017971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:01.017971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:01.017971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:01.017971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:01.017971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:01.017971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:01.017971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:01.017971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:01.019983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:01.019983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:01.019983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:01.019983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:01.019983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:01.019983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:01.019983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:01.019983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:01.019983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:01.019983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:01.019983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:01.019983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:01.020734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c4abae-3bff-4a77-b350-5188cc3807c0 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:01.020734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c4abae-3bff-4a77-b350-5188cc3807c0 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:01.020884 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 244/486] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:01.020963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c4abae-3bff-4a77-b350-5188cc3807c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:01.026195 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5c90ad-b75b-40b4-956d-9e59667568a9 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:01.049433 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:01.055739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-61c4abae-3bff-4a77-b350-5188cc3807c0 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:01.056930 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-61c4abae-3bff-4a77-b350-5188cc3807c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:01.083749 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:01.133213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 243/487] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:01.134203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c4abae-3bff-4a77-b350-5188cc3807c0 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:01.168793 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5c90ad-b75b-40b4-956d-9e59667568a9 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:01.204291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4a47b4-8444-4d6e-91f5-283b3d1e8874 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:01.204815 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4a47b4-8444-4d6e-91f5-283b3d1e8874 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:01.205107 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4a47b4-8444-4d6e-91f5-283b3d1e8874 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:01.205430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4a47b4-8444-4d6e-91f5-283b3d1e8874 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:01.206156 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4a47b4-8444-4d6e-91f5-283b3d1e8874 None admin] RBAC: Adding query filter params (name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:01.207154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4a47b4-8444-4d6e-91f5-283b3d1e8874 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:01.208738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4a47b4-8444-4d6e-91f5-283b3d1e8874 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:01.217578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 245/488] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:01.218947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98377949-11ba-4e69-a9c8-20616e542b36 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:01.219471 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98377949-11ba-4e69-a9c8-20616e542b36 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:01.219471 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98377949-11ba-4e69-a9c8-20616e542b36 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:01.220927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 246/489] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:01.225542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98377949-11ba-4e69-a9c8-20616e542b36 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:01.248260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c4abae-3bff-4a77-b350-5188cc3807c0 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:01.285338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc186c9-8935-42af-9a0c-efe600ec836a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:01.286207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc186c9-8935-42af-9a0c-efe600ec836a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:01.286665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc186c9-8935-42af-9a0c-efe600ec836a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:01.287087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc186c9-8935-42af-9a0c-efe600ec836a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:01.290204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:01.300511 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc186c9-8935-42af-9a0c-efe600ec836a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:01.305791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-fdc186c9-8935-42af-9a0c-efe600ec836a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:01.305791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:01.305791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:01.305791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:01.305791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:01.305791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:01.305791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:01.305791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:01.305791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:01.305791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:01.305791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:01.305791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:01.305791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:01.305791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:01.305791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:01.305791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:01.315343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:01.315343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:01.315343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:01.315343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:01.315343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:01.315343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:01.315343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:01.315343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:01.315343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:01.315343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:01.315343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:01.315343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:01.315343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:01.315343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:01.315343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:01.315343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:01.317046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:01.317046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:01.317046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:01.317046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:01.317046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:01.317046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:01.317046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:01.317046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:01.317046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:01.317046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:01.317046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:01.317046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:01.317046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 244/490] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:01.317046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ff2e7a-44d2-40c9-bcf6-358383f2bbe1 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:01.317905 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ff2e7a-44d2-40c9-bcf6-358383f2bbe1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:01.317905 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ff2e7a-44d2-40c9-bcf6-358383f2bbe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:01.333538 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-85ff2e7a-44d2-40c9-bcf6-358383f2bbe1 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:01.334963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-85ff2e7a-44d2-40c9-bcf6-358383f2bbe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:01.336697 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:01.344517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98377949-11ba-4e69-a9c8-20616e542b36 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:01.381686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6fc1af-6308-48ea-abbb-b4b2ac4e7832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:01.382594 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6fc1af-6308-48ea-abbb-b4b2ac4e7832 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:01.383039 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6fc1af-6308-48ea-abbb-b4b2ac4e7832 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:01.383617 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6fc1af-6308-48ea-abbb-b4b2ac4e7832 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:01.395405 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6fc1af-6308-48ea-abbb-b4b2ac4e7832 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:01.397095 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6fc1af-6308-48ea-abbb-b4b2ac4e7832 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:01.412620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-1b6fc1af-6308-48ea-abbb-b4b2ac4e7832 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:01.412620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:01.412620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:01.412620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:01.412620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:01.412620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:01.412620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:01.412620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:01.412620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:01.412620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:01.412620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:01.412620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:01.412620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:01.412620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:01.412620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:01.412620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:01.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:01.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:01.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:01.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:01.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:01.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:01.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:01.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:01.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:01.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:01.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:01.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:01.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:01.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:01.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:01.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:01.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:01.414737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:01.414737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:01.414737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:01.421126 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 245/492] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:01.421258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 247/492] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:01.421331 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ff2e7a-44d2-40c9-bcf6-358383f2bbe1 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:01.421400 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6fc1af-6308-48ea-abbb-b4b2ac4e7832 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:01.539648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ff2e7a-44d2-40c9-bcf6-358383f2bbe1 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:01.566551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6fc1af-6308-48ea-abbb-b4b2ac4e7832 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:01.567489 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:01.577032 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f181b86-b7ab-4d2f-9157-e09de47708b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:01.581487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f181b86-b7ab-4d2f-9157-e09de47708b4 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:01.581487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f181b86-b7ab-4d2f-9157-e09de47708b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:01.581487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f181b86-b7ab-4d2f-9157-e09de47708b4 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:01.581487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f181b86-b7ab-4d2f-9157-e09de47708b4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:01.581487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f181b86-b7ab-4d2f-9157-e09de47708b4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:01.581487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f181b86-b7ab-4d2f-9157-e09de47708b4 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:01.589918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 246/493] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:01.590746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f181b86-b7ab-4d2f-9157-e09de47708b4 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:01.606307 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2586e4d6-9497-464f-8da9-a41adee321e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:01.607560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2586e4d6-9497-464f-8da9-a41adee321e5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:01.608568 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2586e4d6-9497-464f-8da9-a41adee321e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:01.609213 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2586e4d6-9497-464f-8da9-a41adee321e5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:01.625255 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2586e4d6-9497-464f-8da9-a41adee321e5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:01.633564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-2586e4d6-9497-464f-8da9-a41adee321e5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:01.633564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:01.633564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:01.633564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:01.633564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:01.633564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:01.633564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:01.633564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:01.633564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:01.633564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:01.633564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:01.633564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:01.633564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:01.633564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:01.633564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:01.633564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:01.634616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:01.634616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:01.634616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:01.634616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:01.634616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:01.634616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:01.634616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:01.634616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:01.634616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:01.634616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:01.634616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:01.634616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:01.634616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:01.634616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:01.634616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:01.634616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:01.635572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:01.635572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:01.635572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:01.635572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:01.635572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:01.635572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:01.635572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:01.635572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:01.635572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:01.635572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:01.635572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:01.635572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:01.637840 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 248/494] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:01.639360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2586e4d6-9497-464f-8da9-a41adee321e5 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:01.652156 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:01.717090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f181b86-b7ab-4d2f-9157-e09de47708b4 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:01.753711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2586e4d6-9497-464f-8da9-a41adee321e5 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:01.755254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e793e1-5ccc-42e2-9a7f-a39896779610 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:01.755883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e793e1-5ccc-42e2-9a7f-a39896779610 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:01.756952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e793e1-5ccc-42e2-9a7f-a39896779610 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:01.757264 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e793e1-5ccc-42e2-9a7f-a39896779610 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:01.757925 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e793e1-5ccc-42e2-9a7f-a39896779610 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:01.758231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e793e1-5ccc-42e2-9a7f-a39896779610 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:01.759596 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e793e1-5ccc-42e2-9a7f-a39896779610 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:01.776811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 247/495] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:01.776811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e793e1-5ccc-42e2-9a7f-a39896779610 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:01.792374 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817d4186-f522-4578-9334-bac4aae5e91b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:01.793010 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817d4186-f522-4578-9334-bac4aae5e91b None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:01.793328 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817d4186-f522-4578-9334-bac4aae5e91b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:01.793605 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817d4186-f522-4578-9334-bac4aae5e91b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:01.804772 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817d4186-f522-4578-9334-bac4aae5e91b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:01.806275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817d4186-f522-4578-9334-bac4aae5e91b None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:01.821185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-817d4186-f522-4578-9334-bac4aae5e91b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:01.821185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:01.821185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:01.821185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:01.821185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:01.821185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:01.821185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:01.821185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:01.821185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:01.821185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:01.821185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:01.821185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:01.821185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:01.821185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:01.821185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:01.821185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:01.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:01.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:01.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:01.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:01.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:01.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:01.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:01.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:01.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:01.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:01.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:01.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:01.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:01.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:01.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:01.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:01.822077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:01.827998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:01.827998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:01.827998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:01.827998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 249/496] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:01.827998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817d4186-f522-4578-9334-bac4aae5e91b None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:01.835622 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:01.889991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:01.890510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e793e1-5ccc-42e2-9a7f-a39896779610 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:01.934725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99693eb-ff26-4c95-8a81-ed58a13eb84d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:01.935376 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99693eb-ff26-4c95-8a81-ed58a13eb84d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:01.935662 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99693eb-ff26-4c95-8a81-ed58a13eb84d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:01.935935 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99693eb-ff26-4c95-8a81-ed58a13eb84d None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:01.936612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99693eb-ff26-4c95-8a81-ed58a13eb84d None admin] RBAC: Adding query filter params (name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:01.936919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99693eb-ff26-4c95-8a81-ed58a13eb84d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:01.938343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99693eb-ff26-4c95-8a81-ed58a13eb84d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:01.947509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 248/497] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:01.948432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99693eb-ff26-4c95-8a81-ed58a13eb84d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:01.955906 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817d4186-f522-4578-9334-bac4aae5e91b None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:01.994150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3dee6ab-30bd-43e7-a094-e0771d8846b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:01.994785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3dee6ab-30bd-43e7-a094-e0771d8846b2 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:01.995066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3dee6ab-30bd-43e7-a094-e0771d8846b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:01.995340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3dee6ab-30bd-43e7-a094-e0771d8846b2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:02.007608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3dee6ab-30bd-43e7-a094-e0771d8846b2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:02.014902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-c3dee6ab-30bd-43e7-a094-e0771d8846b2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:02.014902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:02.014902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:02.014902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:02.014902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:02.014902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:02.014902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:02.014902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:02.014902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:02.014902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:02.014902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:02.014902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:02.014902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:02.014902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:02.014902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:02.014902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:02.015804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:02.015804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:02.015804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:02.015804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:02.015804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:02.015804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:02.015804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:02.015804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:02.015804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:02.015804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:02.015804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:02.015804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:02.015804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:02.015804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:02.015804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:02.015804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:02.016598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:02.016598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:02.016598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:02.016598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:02.016598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:02.016598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:02.016598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:02.016598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:02.016598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:02.016598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:02.016598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:02.016598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:02.016598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 250/498] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:02.017267 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3dee6ab-30bd-43e7-a094-e0771d8846b2 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:02.060565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99693eb-ff26-4c95-8a81-ed58a13eb84d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:02.072602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:02.096611 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69dc6ec-7c0a-45ee-b125-90303e4283b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:02.097282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69dc6ec-7c0a-45ee-b125-90303e4283b0 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:02.097571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69dc6ec-7c0a-45ee-b125-90303e4283b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:02.097920 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69dc6ec-7c0a-45ee-b125-90303e4283b0 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:02.106075 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69dc6ec-7c0a-45ee-b125-90303e4283b0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:02.107813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69dc6ec-7c0a-45ee-b125-90303e4283b0 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:02.122564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-e69dc6ec-7c0a-45ee-b125-90303e4283b0 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 04 00:47:02.122564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:02.122564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:02.122564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:02.122564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:02.122564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:02.122564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:02.122564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:02.122564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:02.122564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:02.122564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:02.122564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:02.122564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:02.122564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:02.122564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:02.122564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:02.123928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:02.124995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:02.124995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:02.124995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:02.124995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:02.124995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:02.124995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:02.124995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:02.124995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:02.124995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 04 00:47:02.124995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:02.125921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 249/499] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:47:01 2022] GET /identity/v3/users/glance => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:02.130339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69dc6ec-7c0a-45ee-b125-90303e4283b0 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:02.132628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3dee6ab-30bd-43e7-a094-e0771d8846b2 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:02.152705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:02.169407 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c8fb4d-2601-4090-be40-53dd131448d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:02.170010 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c8fb4d-2601-4090-be40-53dd131448d5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:02.170291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c8fb4d-2601-4090-be40-53dd131448d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:02.170565 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c8fb4d-2601-4090-be40-53dd131448d5 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:02.171200 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c8fb4d-2601-4090-be40-53dd131448d5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:02.171504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c8fb4d-2601-4090-be40-53dd131448d5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:02.172868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c8fb4d-2601-4090-be40-53dd131448d5 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:02.184581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 251/500] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:02.185392 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c8fb4d-2601-4090-be40-53dd131448d5 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:02.234413 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69dc6ec-7c0a-45ee-b125-90303e4283b0 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:02.268226 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171ed1f0-5075-46fd-8292-9a309ceeed1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:02.268868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171ed1f0-5075-46fd-8292-9a309ceeed1e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:02.269204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171ed1f0-5075-46fd-8292-9a309ceeed1e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:02.269477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171ed1f0-5075-46fd-8292-9a309ceeed1e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:02.279633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171ed1f0-5075-46fd-8292-9a309ceeed1e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:02.281068 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171ed1f0-5075-46fd-8292-9a309ceeed1e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:02.284222 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c8fb4d-2601-4090-be40-53dd131448d5 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:02.294742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-171ed1f0-5075-46fd-8292-9a309ceeed1e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:02.294742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:02.294742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:02.294742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:02.294742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:02.294742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:02.294742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:02.294742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:02.294742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:02.294742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:02.294742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:02.294742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:02.294742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:02.294742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:02.294742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:02.294742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:02.300907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:02.300907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:02.300907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:02.300907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:02.300907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:02.300907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:02.300907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:02.300907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:02.300907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:02.300907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:02.300907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:02.300907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:02.300907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:02.300907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:02.300907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:02.300907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:02.300907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:02.302029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:02.302029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:02.302029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:02.302029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 250/501] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:02.302029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171ed1f0-5075-46fd-8292-9a309ceeed1e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:02.321336 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:02.326298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48213712-c36c-4e54-892b-720ede624850 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:02.326900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48213712-c36c-4e54-892b-720ede624850 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:02.327178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48213712-c36c-4e54-892b-720ede624850 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:02.327446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48213712-c36c-4e54-892b-720ede624850 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:02.328137 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48213712-c36c-4e54-892b-720ede624850 None admin] RBAC: Adding query filter params (name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:02.328403 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48213712-c36c-4e54-892b-720ede624850 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:02.329808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48213712-c36c-4e54-892b-720ede624850 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:02.338226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 252/502] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:02.339461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e538a6d0-6487-4dee-8f0e-e6fb677ef505 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:02.339803 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e538a6d0-6487-4dee-8f0e-e6fb677ef505 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:02.340079 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e538a6d0-6487-4dee-8f0e-e6fb677ef505 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:02.341348 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 253/503] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:02.342039 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e538a6d0-6487-4dee-8f0e-e6fb677ef505 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:02.386458 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:02.405355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171ed1f0-5075-46fd-8292-9a309ceeed1e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:02.439207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a15ea2a-c7bc-43e7-aa64-afad78a5e8b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:02.439828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a15ea2a-c7bc-43e7-aa64-afad78a5e8b2 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:02.440105 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a15ea2a-c7bc-43e7-aa64-afad78a5e8b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:02.440375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a15ea2a-c7bc-43e7-aa64-afad78a5e8b2 None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:02.441186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a15ea2a-c7bc-43e7-aa64-afad78a5e8b2 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:02.441496 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a15ea2a-c7bc-43e7-aa64-afad78a5e8b2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:02.442930 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a15ea2a-c7bc-43e7-aa64-afad78a5e8b2 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:02.448732 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e538a6d0-6487-4dee-8f0e-e6fb677ef505 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:02.479903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 251/504] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Sep 4 00:47:02 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:02.480640 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a15ea2a-c7bc-43e7-aa64-afad78a5e8b2 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:02.484387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ecf797-5c65-484b-877b-f09f39caefc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:02.485048 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ecf797-5c65-484b-877b-f09f39caefc7 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:02.485366 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ecf797-5c65-484b-877b-f09f39caefc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:02.485639 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ecf797-5c65-484b-877b-f09f39caefc7 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:02.494914 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ecf797-5c65-484b-877b-f09f39caefc7 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:02.496364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ecf797-5c65-484b-877b-f09f39caefc7 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:02.509504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-e1ecf797-5c65-484b-877b-f09f39caefc7 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:47:02.509504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:02.509504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:02.509504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:02.509504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:02.509504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:02.509504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:02.509504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:02.509504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:02.509504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:02.509504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:02.509504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:02.509504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:02.509504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:02.509504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:02.509504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:02.510430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:02.511298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:02.511298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:02.511298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:02.511298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:02.511298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:02.511298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:02.511298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:02.511298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:02.511298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:47:02.511298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:02.511298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 254/505] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 4 00:47:02 2022] GET /identity/v3/users/nova => generated 82 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:02.511844 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ecf797-5c65-484b-877b-f09f39caefc7 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:02.565147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:02.583917 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a15ea2a-c7bc-43e7-aa64-afad78a5e8b2 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:02.607570 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:02.610543 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ecf797-5c65-484b-877b-f09f39caefc7 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:02.617482 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588bb055-691e-46cf-8e5e-271535e69f3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:02.618077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588bb055-691e-46cf-8e5e-271535e69f3b None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:02.618353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588bb055-691e-46cf-8e5e-271535e69f3b None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:02.618624 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588bb055-691e-46cf-8e5e-271535e69f3b None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:02.619265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588bb055-691e-46cf-8e5e-271535e69f3b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:02.619576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588bb055-691e-46cf-8e5e-271535e69f3b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:02.620930 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588bb055-691e-46cf-8e5e-271535e69f3b None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:02.632842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 252/506] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:02.634143 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6758cc13-a89c-4de1-abdb-856e0b31ff3e None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:02.634508 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6758cc13-a89c-4de1-abdb-856e0b31ff3e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:02.634872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6758cc13-a89c-4de1-abdb-856e0b31ff3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:02.649884 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e2dbf9-9d4c-490f-b866-1341b1ac7a17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:02.650504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e2dbf9-9d4c-490f-b866-1341b1ac7a17 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:02.650782 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e2dbf9-9d4c-490f-b866-1341b1ac7a17 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:02.651053 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e2dbf9-9d4c-490f-b866-1341b1ac7a17 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:02.653231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-6758cc13-a89c-4de1-abdb-856e0b31ff3e None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:02.654424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6758cc13-a89c-4de1-abdb-856e0b31ff3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:02.661224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e2dbf9-9d4c-490f-b866-1341b1ac7a17 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:02.662701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e2dbf9-9d4c-490f-b866-1341b1ac7a17 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:02.676655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-c7e2dbf9-9d4c-490f-b866-1341b1ac7a17 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:02.676655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:02.676655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:02.676655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:02.676655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:02.676655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:02.676655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:02.676655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:02.676655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:02.676655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:02.676655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:02.676655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:02.676655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:02.676655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:02.676655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:02.676655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:02.677679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:02.677679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:02.677679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:02.677679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:02.677679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:02.677679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:02.677679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:02.677679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:02.677679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:02.677679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:02.677679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:02.677679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:02.677679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:02.677679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:02.677679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:02.677679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:02.677679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:02.678662 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:02.678662 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:02.678662 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:02.678662 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 255/507] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:02.678662 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e2dbf9-9d4c-490f-b866-1341b1ac7a17 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:02.722611 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 253/508] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:02.723440 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6758cc13-a89c-4de1-abdb-856e0b31ff3e None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:02.789709 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e2dbf9-9d4c-490f-b866-1341b1ac7a17 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:02.790022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:02.821671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6758cc13-a89c-4de1-abdb-856e0b31ff3e None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:02.823763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fc8344-2f87-4912-93e4-e2945abcdcb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:02.824359 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fc8344-2f87-4912-93e4-e2945abcdcb0 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:02.824635 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fc8344-2f87-4912-93e4-e2945abcdcb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:02.824905 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fc8344-2f87-4912-93e4-e2945abcdcb0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:02.849304 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:02.856711 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c310b3fb-9035-4a3e-8808-b6a3157a7195 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:02.862689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c310b3fb-9035-4a3e-8808-b6a3157a7195 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:02.862689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c310b3fb-9035-4a3e-8808-b6a3157a7195 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:02.862689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c310b3fb-9035-4a3e-8808-b6a3157a7195 None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:02.862689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c310b3fb-9035-4a3e-8808-b6a3157a7195 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:02.862689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c310b3fb-9035-4a3e-8808-b6a3157a7195 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:02.862689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c310b3fb-9035-4a3e-8808-b6a3157a7195 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:02.880849 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 254/509] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 4 00:47:02 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:02.881921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c310b3fb-9035-4a3e-8808-b6a3157a7195 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:02.911288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fc8344-2f87-4912-93e4-e2945abcdcb0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:02.913093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fc8344-2f87-4912-93e4-e2945abcdcb0 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:02.984077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c310b3fb-9035-4a3e-8808-b6a3157a7195 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:03.015131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:03.023056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb041bb-2f9b-4adb-a989-25e4910281bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:03.023690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb041bb-2f9b-4adb-a989-25e4910281bc None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:03.023986 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb041bb-2f9b-4adb-a989-25e4910281bc None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:03.024253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb041bb-2f9b-4adb-a989-25e4910281bc None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:03.025466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 256/510] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:47:03.026307 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fc8344-2f87-4912-93e4-e2945abcdcb0 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:03.031584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb041bb-2f9b-4adb-a989-25e4910281bc None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:03.033029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb041bb-2f9b-4adb-a989-25e4910281bc None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:03.045334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-dbb041bb-2f9b-4adb-a989-25e4910281bc None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 04 00:47:03.045334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:03.045334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:03.045334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:03.045334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:03.045334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:03.045334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:03.045334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:03.045334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:03.045334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:03.045334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:03.045334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:03.045334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:03.045334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:03.045334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:03.045334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:03.046241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:03.047161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:03.047161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:03.047161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:03.047161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:03.047161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:03.047161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:03.047161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:03.047161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:03.047161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 04 00:47:03.047161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:03.049224 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 255/511] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:02 2022] GET /identity/v3/users/neutron => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:03.051209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb041bb-2f9b-4adb-a989-25e4910281bc None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:03.080699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:03.126506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fc8344-2f87-4912-93e4-e2945abcdcb0 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:03.152242 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb041bb-2f9b-4adb-a989-25e4910281bc None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:03.160902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86447b6-8223-45b5-95c5-4e10054a614b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:03.161536 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86447b6-8223-45b5-95c5-4e10054a614b None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:03.161816 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86447b6-8223-45b5-95c5-4e10054a614b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:03.162088 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86447b6-8223-45b5-95c5-4e10054a614b None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:03.162725 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86447b6-8223-45b5-95c5-4e10054a614b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:03.163025 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86447b6-8223-45b5-95c5-4e10054a614b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:03.164400 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e86447b6-8223-45b5-95c5-4e10054a614b None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:03.176080 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 257/512] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:03.177889 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86447b6-8223-45b5-95c5-4e10054a614b None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:03.186075 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4a38f6-e0c9-46e4-9da9-337aeeb7f94f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:03.186681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4a38f6-e0c9-46e4-9da9-337aeeb7f94f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:03.186964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4a38f6-e0c9-46e4-9da9-337aeeb7f94f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:03.187237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4a38f6-e0c9-46e4-9da9-337aeeb7f94f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:03.242890 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:03.255819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4a38f6-e0c9-46e4-9da9-337aeeb7f94f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:03.257303 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4a38f6-e0c9-46e4-9da9-337aeeb7f94f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:03.277926 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86447b6-8223-45b5-95c5-4e10054a614b None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:03.290839 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:03.314343 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a6a69c-1c2a-4bff-a4ea-d0356d985a1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:03.315233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a6a69c-1c2a-4bff-a4ea-d0356d985a1a None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:03.316261 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a6a69c-1c2a-4bff-a4ea-d0356d985a1a None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:03.316672 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a6a69c-1c2a-4bff-a4ea-d0356d985a1a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:03.332771 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a6a69c-1c2a-4bff-a4ea-d0356d985a1a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:03.335244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a6a69c-1c2a-4bff-a4ea-d0356d985a1a None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:03.351609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-37a6a69c-1c2a-4bff-a4ea-d0356d985a1a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:03.351609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:03.351609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:03.351609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:03.351609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:03.351609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:03.351609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:03.351609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:03.351609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:03.351609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:03.351609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:03.351609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:03.351609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:03.351609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:03.351609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:03.351609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:03.353067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:03.353067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:03.353067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:03.353067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:03.353067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:03.353067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:03.353067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:03.353067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:03.353067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:03.353067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:03.353067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:03.353067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:03.353067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:03.353067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:03.353067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:03.353067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:03.353067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:03.353939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:03.353939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:03.353939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:03.353939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 258/513] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:03.353939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a6a69c-1c2a-4bff-a4ea-d0356d985a1a None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:03.365093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 256/514] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:47:03.365093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4a38f6-e0c9-46e4-9da9-337aeeb7f94f None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:03.461798 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:03.479581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4a38f6-e0c9-46e4-9da9-337aeeb7f94f None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:03.516367 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a6a69c-1c2a-4bff-a4ea-d0356d985a1a None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:03.520700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2573f6-17db-4846-aa29-ae83f6659f0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:03.521323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2573f6-17db-4846-aa29-ae83f6659f0e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:03.521604 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2573f6-17db-4846-aa29-ae83f6659f0e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:03.521873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2573f6-17db-4846-aa29-ae83f6659f0e None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:03.522595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2573f6-17db-4846-aa29-ae83f6659f0e None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:03.522897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2573f6-17db-4846-aa29-ae83f6659f0e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:03.524314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2573f6-17db-4846-aa29-ae83f6659f0e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:03.542409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 257/515] 10.222.0.42 () {56 vars in 1139 bytes} [Sun Sep 4 00:47:03 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:03.543308 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2573f6-17db-4846-aa29-ae83f6659f0e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:03.550998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed6de4c-7622-4cc1-922a-2fc5aafee5ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:03.551628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed6de4c-7622-4cc1-922a-2fc5aafee5ca None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:03.551909 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed6de4c-7622-4cc1-922a-2fc5aafee5ca None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:03.552179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed6de4c-7622-4cc1-922a-2fc5aafee5ca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:03.562564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed6de4c-7622-4cc1-922a-2fc5aafee5ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:03.564006 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed6de4c-7622-4cc1-922a-2fc5aafee5ca None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:03.577806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-bed6de4c-7622-4cc1-922a-2fc5aafee5ca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:03.577806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:03.577806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:03.577806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:03.577806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:03.577806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:03.577806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:03.577806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:03.577806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:03.577806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:03.577806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:03.577806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:03.577806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:03.577806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:03.577806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:03.577806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:03.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:03.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:03.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:03.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:03.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:03.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:03.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:03.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:03.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:03.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:03.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:03.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:03.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:03.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:03.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:03.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:03.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:03.579542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:03.579542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:03.579542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:03.579542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 259/516] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:03.579762 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed6de4c-7622-4cc1-922a-2fc5aafee5ca None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:03.598047 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:03.660157 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2573f6-17db-4846-aa29-ae83f6659f0e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:03.694248 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed6de4c-7622-4cc1-922a-2fc5aafee5ca None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:03.699764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32a21ae-d0ab-433c-a0f4-1c89aa9b6f6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:03.700386 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32a21ae-d0ab-433c-a0f4-1c89aa9b6f6a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:03.700665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32a21ae-d0ab-433c-a0f4-1c89aa9b6f6a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:03.700935 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a32a21ae-d0ab-433c-a0f4-1c89aa9b6f6a None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:03.708473 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32a21ae-d0ab-433c-a0f4-1c89aa9b6f6a None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:03.709929 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a32a21ae-d0ab-433c-a0f4-1c89aa9b6f6a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:03.721108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-a32a21ae-d0ab-433c-a0f4-1c89aa9b6f6a None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 04 00:47:03.721108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:03.721108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:03.721108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:03.721108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:03.721108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:03.721108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:03.721108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:03.721108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:03.721108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:03.721108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:03.721108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:03.721108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:03.721108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:03.721108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:03.721108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:03.722768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:03.723810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:03.723810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:03.723810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:03.723810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:03.723810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:03.723810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:03.723810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:03.723810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:03.723810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 04 00:47:03.723810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:03.723810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 258/517] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:47:03 2022] GET /identity/v3/users/cinder => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:03.723810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32a21ae-d0ab-433c-a0f4-1c89aa9b6f6a None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:03.730308 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f111ae-05fc-4881-a061-b9157d0c1fa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:03.730963 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f111ae-05fc-4881-a061-b9157d0c1fa6 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:03.731238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f111ae-05fc-4881-a061-b9157d0c1fa6 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:03.731509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f111ae-05fc-4881-a061-b9157d0c1fa6 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:03.731919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:03.732211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f111ae-05fc-4881-a061-b9157d0c1fa6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:03.732513 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f111ae-05fc-4881-a061-b9157d0c1fa6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:03.733904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f111ae-05fc-4881-a061-b9157d0c1fa6 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:03.745164 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 260/518] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:03.745164 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f111ae-05fc-4881-a061-b9157d0c1fa6 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:03.821534 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a32a21ae-d0ab-433c-a0f4-1c89aa9b6f6a None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:03.822726 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:03.845034 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f111ae-05fc-4881-a061-b9157d0c1fa6 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:03.855106 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35066ecf-8a21-4e79-9583-eb07ee124dc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:03.855664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35066ecf-8a21-4e79-9583-eb07ee124dc8 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:03.855947 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35066ecf-8a21-4e79-9583-eb07ee124dc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:03.856218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35066ecf-8a21-4e79-9583-eb07ee124dc8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:03.867129 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35066ecf-8a21-4e79-9583-eb07ee124dc8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:03.868560 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35066ecf-8a21-4e79-9583-eb07ee124dc8 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:03.880015 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb4f2d5-6015-4e1a-8246-32eda5c0ee65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:03.880606 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb4f2d5-6015-4e1a-8246-32eda5c0ee65 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:03.880889 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb4f2d5-6015-4e1a-8246-32eda5c0ee65 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:03.881185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb4f2d5-6015-4e1a-8246-32eda5c0ee65 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:03.883260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-35066ecf-8a21-4e79-9583-eb07ee124dc8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:03.883260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:03.883260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:03.883260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:03.883260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:03.883260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:03.883260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:03.883260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:03.883260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:03.883260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:03.883260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:03.883260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:03.883260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:03.883260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:03.883260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:03.883260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:03.884109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:03.884109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:03.884109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:03.884109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:03.884109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:03.884109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:03.884109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:03.884109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:03.884109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:03.884109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:03.884109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:03.884109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:03.884109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:03.884109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:03.884109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:03.884109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:03.884109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:03.884937 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:03.884937 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:03.884937 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:03.888993 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 259/519] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:03.889738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35066ecf-8a21-4e79-9583-eb07ee124dc8 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:03.949025 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:03.955437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb4f2d5-6015-4e1a-8246-32eda5c0ee65 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:03.958666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb4f2d5-6015-4e1a-8246-32eda5c0ee65 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:03.991164 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35066ecf-8a21-4e79-9583-eb07ee124dc8 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:04.024489 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884019e1-d5db-43d4-9b22-28d37c98ddd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:04.025128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884019e1-d5db-43d4-9b22-28d37c98ddd2 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:04.025419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884019e1-d5db-43d4-9b22-28d37c98ddd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:04.025694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884019e1-d5db-43d4-9b22-28d37c98ddd2 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:04.026338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884019e1-d5db-43d4-9b22-28d37c98ddd2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:04.026642 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884019e1-d5db-43d4-9b22-28d37c98ddd2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:04.028005 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884019e1-d5db-43d4-9b22-28d37c98ddd2 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:04.033922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:04.040289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 260/520] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:04.041123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884019e1-d5db-43d4-9b22-28d37c98ddd2 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:04.062464 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 261/521] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:47:04.063283 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb4f2d5-6015-4e1a-8246-32eda5c0ee65 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:04.141683 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884019e1-d5db-43d4-9b22-28d37c98ddd2 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:04.158691 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:04.162534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb4f2d5-6015-4e1a-8246-32eda5c0ee65 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:04.174781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce9cfbb-a874-4158-bfe8-a55bbc593d53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:04.175479 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce9cfbb-a874-4158-bfe8-a55bbc593d53 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:04.175866 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce9cfbb-a874-4158-bfe8-a55bbc593d53 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:04.176140 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce9cfbb-a874-4158-bfe8-a55bbc593d53 None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:04.176873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce9cfbb-a874-4158-bfe8-a55bbc593d53 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:04.177201 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce9cfbb-a874-4158-bfe8-a55bbc593d53 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:04.178696 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce9cfbb-a874-4158-bfe8-a55bbc593d53 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:04.196521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 261/522] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Sep 4 00:47:04 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:04.197350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce9cfbb-a874-4158-bfe8-a55bbc593d53 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:04.199404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7fa067-081f-47c6-a7cb-3bd3b4268b25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:04.200182 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7fa067-081f-47c6-a7cb-3bd3b4268b25 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:04.201301 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7fa067-081f-47c6-a7cb-3bd3b4268b25 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:04.201587 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7fa067-081f-47c6-a7cb-3bd3b4268b25 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:04.202235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7fa067-081f-47c6-a7cb-3bd3b4268b25 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:04.203670 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7fa067-081f-47c6-a7cb-3bd3b4268b25 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:04.277637 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 262/523] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:47:04 2022] POST /identity/v3/users => generated 247 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:47:04.278467 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7fa067-081f-47c6-a7cb-3bd3b4268b25 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:04.299321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce9cfbb-a874-4158-bfe8-a55bbc593d53 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:04.329874 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:04.341911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3102d022-777f-4c28-93d0-3740bcebae4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:04.342541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3102d022-777f-4c28-93d0-3740bcebae4b None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:04.342818 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3102d022-777f-4c28-93d0-3740bcebae4b None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:04.343087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3102d022-777f-4c28-93d0-3740bcebae4b None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:04.343779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3102d022-777f-4c28-93d0-3740bcebae4b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:04.344109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3102d022-777f-4c28-93d0-3740bcebae4b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:04.345595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3102d022-777f-4c28-93d0-3740bcebae4b None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:04.358896 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 262/524] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:04.359776 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3102d022-777f-4c28-93d0-3740bcebae4b None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:04.400861 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:04.420108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7fa067-081f-47c6-a7cb-3bd3b4268b25 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:04.464150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4308c999-4cea-4cf7-95d1-26267fd4af3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:04.464944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4308c999-4cea-4cf7-95d1-26267fd4af3a None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:04.465485 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4308c999-4cea-4cf7-95d1-26267fd4af3a None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:04.465897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4308c999-4cea-4cf7-95d1-26267fd4af3a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:04.473477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4308c999-4cea-4cf7-95d1-26267fd4af3a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:04.474918 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4308c999-4cea-4cf7-95d1-26267fd4af3a None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:04.480858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3102d022-777f-4c28-93d0-3740bcebae4b None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:04.488226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-4308c999-4cea-4cf7-95d1-26267fd4af3a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:04.488226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:04.488226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:04.488226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:04.488226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:04.488226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:04.488226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:04.488226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:04.488226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:04.488226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:04.488226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:04.488226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:04.488226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:04.488226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:04.488226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:04.489335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:04.489335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:04.489335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:04.489335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:04.489335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:04.489335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:04.489335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:04.489335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:04.489335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:04.489335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:04.489335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:04.489335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:04.489335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:04.489335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:04.489335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:04.489335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:04.489335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:04.490384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:04.490384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:04.490384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:04.490384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:04.490384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:04.490384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:04.490384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:04.490384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:04.490384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 263/525] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:04.490384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4308c999-4cea-4cf7-95d1-26267fd4af3a None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:04.523729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ade03a6-14e9-4036-bf93-114a42939951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:04.524383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ade03a6-14e9-4036-bf93-114a42939951 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:04.524899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ade03a6-14e9-4036-bf93-114a42939951 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:04.525208 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ade03a6-14e9-4036-bf93-114a42939951 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:04.538404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ade03a6-14e9-4036-bf93-114a42939951 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:04.539846 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ade03a6-14e9-4036-bf93-114a42939951 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:04.555218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-9ade03a6-14e9-4036-bf93-114a42939951 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:04.555218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:04.555218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:04.555218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:04.555218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:04.555218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:04.555218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:04.555218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:04.555218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:04.555218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:04.555218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:04.555218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:04.555218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:04.555218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:04.555218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:04.555218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:04.556231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:04.556231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:04.556231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:04.556231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:04.556231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:04.556231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:04.556231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:04.556231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:04.556231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:04.556231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:04.556231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:04.556231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:04.556231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:04.556231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:04.556231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:04.556231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:04.556231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:04.557061 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:04.557061 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:04.557061 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:04.557061 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 263/526] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:04.557324 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ade03a6-14e9-4036-bf93-114a42939951 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:04.598806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:04.608138 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4308c999-4cea-4cf7-95d1-26267fd4af3a None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:04.646887 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2c650a-869d-4237-ba23-1781151c9a01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:04.647494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2c650a-869d-4237-ba23-1781151c9a01 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:04.647776 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2c650a-869d-4237-ba23-1781151c9a01 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:04.648052 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2c650a-869d-4237-ba23-1781151c9a01 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:04.669209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:04.682115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ade03a6-14e9-4036-bf93-114a42939951 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:04.717591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2c650a-869d-4237-ba23-1781151c9a01 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:04.718544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d40d1d4-9c90-44f8-9b1a-b745272d3bc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:04.719047 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2c650a-869d-4237-ba23-1781151c9a01 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:04.719218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d40d1d4-9c90-44f8-9b1a-b745272d3bc8 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:04.719448 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d40d1d4-9c90-44f8-9b1a-b745272d3bc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:04.719720 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d40d1d4-9c90-44f8-9b1a-b745272d3bc8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:04.727576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d40d1d4-9c90-44f8-9b1a-b745272d3bc8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:04.729489 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d40d1d4-9c90-44f8-9b1a-b745272d3bc8 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:04.742933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-8d40d1d4-9c90-44f8-9b1a-b745272d3bc8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:04.742933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:04.742933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:04.742933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:04.742933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:04.742933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:04.742933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:04.742933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:04.742933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:04.742933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:04.742933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:04.742933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:04.742933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:04.742933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:04.742933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:04.743795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:04.743795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:04.743795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:04.743795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:04.743795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:04.743795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:04.743795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:04.743795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:04.743795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:04.743795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:04.743795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:04.743795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:04.743795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:04.743795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:04.743795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:04.743795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:04.743795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:04.744671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:04.744671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:04.744671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:04.744671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:04.744671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:04.744671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:04.744671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:04.744671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:04.744671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 264/527] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:04.745914 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d40d1d4-9c90-44f8-9b1a-b745272d3bc8 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:04.823210 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:04.832797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 264/528] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:47:04.833654 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2c650a-869d-4237-ba23-1781151c9a01 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:04.860025 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d40d1d4-9c90-44f8-9b1a-b745272d3bc8 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:04.895396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa471cfb-3cfd-4824-acc0-0f6b51a8497d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:04.895968 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa471cfb-3cfd-4824-acc0-0f6b51a8497d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:04.896244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa471cfb-3cfd-4824-acc0-0f6b51a8497d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:04.896526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa471cfb-3cfd-4824-acc0-0f6b51a8497d None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:04.897261 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa471cfb-3cfd-4824-acc0-0f6b51a8497d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:04.897586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa471cfb-3cfd-4824-acc0-0f6b51a8497d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:04.898996 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa471cfb-3cfd-4824-acc0-0f6b51a8497d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:04.921509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 265/529] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:04.921509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa471cfb-3cfd-4824-acc0-0f6b51a8497d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:04.932169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:04.945313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2c650a-869d-4237-ba23-1781151c9a01 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:04.981536 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b563d4b-7bb7-46a1-8f57-91b6e197443b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:04.982131 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b563d4b-7bb7-46a1-8f57-91b6e197443b None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:04.982411 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b563d4b-7bb7-46a1-8f57-91b6e197443b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:04.982675 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b563d4b-7bb7-46a1-8f57-91b6e197443b None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:04.983307 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b563d4b-7bb7-46a1-8f57-91b6e197443b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:04.983604 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b563d4b-7bb7-46a1-8f57-91b6e197443b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:04.984965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b563d4b-7bb7-46a1-8f57-91b6e197443b None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:05.000885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 265/530] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:05.000885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b563d4b-7bb7-46a1-8f57-91b6e197443b None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:05.030769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa471cfb-3cfd-4824-acc0-0f6b51a8497d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:05.052194 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:05.066692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be46007-910c-4093-bf80-f99038039f86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:05.067293 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be46007-910c-4093-bf80-f99038039f86 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:05.067573 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be46007-910c-4093-bf80-f99038039f86 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:05.067842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be46007-910c-4093-bf80-f99038039f86 None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:05.068528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be46007-910c-4093-bf80-f99038039f86 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:05.068831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be46007-910c-4093-bf80-f99038039f86 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:05.070278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be46007-910c-4093-bf80-f99038039f86 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:05.081816 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 266/531] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:05.082613 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be46007-910c-4093-bf80-f99038039f86 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:05.104743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b563d4b-7bb7-46a1-8f57-91b6e197443b None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:05.142512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22a249e-eb6a-4e3b-838f-e504851335ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:05.143445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22a249e-eb6a-4e3b-838f-e504851335ea None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:05.144185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22a249e-eb6a-4e3b-838f-e504851335ea None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:05.144606 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22a249e-eb6a-4e3b-838f-e504851335ea None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:05.152394 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:05.156654 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22a249e-eb6a-4e3b-838f-e504851335ea None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:05.158184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22a249e-eb6a-4e3b-838f-e504851335ea None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:05.173030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-b22a249e-eb6a-4e3b-838f-e504851335ea None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:05.173030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:05.173030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:05.173030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:05.173030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:05.173030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:05.173030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:05.173030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:05.173030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:05.173030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:05.173030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:05.173030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:05.173030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:05.173030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:05.173030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:05.173030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:05.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:05.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:05.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:05.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:05.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:05.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:05.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:05.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:05.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:05.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:05.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:05.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:05.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:05.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:05.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:05.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:05.173975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:05.174809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:05.174809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:05.174809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:05.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 266/532] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:05.179586 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22a249e-eb6a-4e3b-838f-e504851335ea None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:05.187540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be46007-910c-4093-bf80-f99038039f86 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:05.222375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db0a8f4-cd6a-48d1-b1b5-5b3f96eef98a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:05.222982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db0a8f4-cd6a-48d1-b1b5-5b3f96eef98a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:05.223267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db0a8f4-cd6a-48d1-b1b5-5b3f96eef98a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:05.223539 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db0a8f4-cd6a-48d1-b1b5-5b3f96eef98a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:05.224264 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db0a8f4-cd6a-48d1-b1b5-5b3f96eef98a None admin] RBAC: Adding query filter params (role.id=df912a77fbaa479db172f76a214f8bc8, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=69153344fc4b45588bdbcecdfcfd7c3f) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:05.224577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db0a8f4-cd6a-48d1-b1b5-5b3f96eef98a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:05.226069 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db0a8f4-cd6a-48d1-b1b5-5b3f96eef98a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:05.238033 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 267/533] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:05 2022] GET /identity/v3/role_assignments?user.id=69153344fc4b45588bdbcecdfcfd7c3f&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=df912a77fbaa479db172f76a214f8bc8 => generated 259 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:05.239366 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db0a8f4-cd6a-48d1-b1b5-5b3f96eef98a None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:05.295768 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22a249e-eb6a-4e3b-838f-e504851335ea None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:05.299058 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:05.338204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c1ec64-938d-4eb5-bb35-bad0a7f73744 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:05.339018 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c1ec64-938d-4eb5-bb35-bad0a7f73744 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:05.339397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c1ec64-938d-4eb5-bb35-bad0a7f73744 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:05.339766 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c1ec64-938d-4eb5-bb35-bad0a7f73744 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:05.348123 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c1ec64-938d-4eb5-bb35-bad0a7f73744 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:05.349890 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c1ec64-938d-4eb5-bb35-bad0a7f73744 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:05.360265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db0a8f4-cd6a-48d1-b1b5-5b3f96eef98a None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:05.364345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-44c1ec64-938d-4eb5-bb35-bad0a7f73744 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:05.364345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:05.364345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:05.364345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:05.364345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:05.364345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:05.364345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:05.364345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:05.364345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:05.364345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:05.364345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:05.364345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:05.364345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:05.364345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:05.364345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:05.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:05.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:05.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:05.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:05.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:05.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:05.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:05.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:05.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:05.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:05.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:05.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:05.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:05.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:05.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:05.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:05.365292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:05.366334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:05.366334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:05.366334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:05.366334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:05.366334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:05.366334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:05.366334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:05.366334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:05.367215 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 267/534] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:05.368074 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c1ec64-938d-4eb5-bb35-bad0a7f73744 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:05.398019 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:05.403953 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2ce66f-ac0e-4408-9758-c2d8e8ad69b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:05.404653 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2ce66f-ac0e-4408-9758-c2d8e8ad69b0 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:05.404842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2ce66f-ac0e-4408-9758-c2d8e8ad69b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:05.405103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2ce66f-ac0e-4408-9758-c2d8e8ad69b0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:05.405919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2ce66f-ac0e-4408-9758-c2d8e8ad69b0 None admin] RBAC: Adding query filter params (role.id=df912a77fbaa479db172f76a214f8bc8, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=57c88f16c7df40adb917f49804fda21f) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:05.406990 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2ce66f-ac0e-4408-9758-c2d8e8ad69b0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:05.407684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2ce66f-ac0e-4408-9758-c2d8e8ad69b0 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:05.423922 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 268/535] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:05 2022] GET /identity/v3/role_assignments?user.id=57c88f16c7df40adb917f49804fda21f&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=df912a77fbaa479db172f76a214f8bc8 => generated 259 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:05.426277 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2ce66f-ac0e-4408-9758-c2d8e8ad69b0 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:05.517610 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c1ec64-938d-4eb5-bb35-bad0a7f73744 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:05.558261 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d0bae4-4799-41d8-a83f-ebaf8c87b724 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:05.558909 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d0bae4-4799-41d8-a83f-ebaf8c87b724 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:05.559191 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d0bae4-4799-41d8-a83f-ebaf8c87b724 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:05.559462 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d0bae4-4799-41d8-a83f-ebaf8c87b724 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:05.560117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d0bae4-4799-41d8-a83f-ebaf8c87b724 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:05.560419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d0bae4-4799-41d8-a83f-ebaf8c87b724 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:05.561838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d0bae4-4799-41d8-a83f-ebaf8c87b724 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:05.575147 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 268/536] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:05.580531 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d0bae4-4799-41d8-a83f-ebaf8c87b724 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:05.616535 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:05.643140 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2ce66f-ac0e-4408-9758-c2d8e8ad69b0 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:05.703552 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d0bae4-4799-41d8-a83f-ebaf8c87b724 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:05.719153 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0586fb44-8b7a-4b2b-b1f7-4a0ac1a25709 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:05.719767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0586fb44-8b7a-4b2b-b1f7-4a0ac1a25709 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:05.720052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0586fb44-8b7a-4b2b-b1f7-4a0ac1a25709 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:05.720321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0586fb44-8b7a-4b2b-b1f7-4a0ac1a25709 None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:05.721023 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0586fb44-8b7a-4b2b-b1f7-4a0ac1a25709 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:05.721408 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0586fb44-8b7a-4b2b-b1f7-4a0ac1a25709 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:05.722914 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0586fb44-8b7a-4b2b-b1f7-4a0ac1a25709 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:05.741584 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadf4e60-df13-4262-a2ca-f01469528f61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:05.742571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadf4e60-df13-4262-a2ca-f01469528f61 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:05.743044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadf4e60-df13-4262-a2ca-f01469528f61 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:05.743522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadf4e60-df13-4262-a2ca-f01469528f61 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:05.751071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadf4e60-df13-4262-a2ca-f01469528f61 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:05.751944 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 269/537] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:05.753635 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadf4e60-df13-4262-a2ca-f01469528f61 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:05.758734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0586fb44-8b7a-4b2b-b1f7-4a0ac1a25709 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:05.766117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-dadf4e60-df13-4262-a2ca-f01469528f61 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:05.766117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:05.766117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:05.766117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:05.766117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:05.766117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:05.766117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:05.766117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:05.766117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:05.766117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:05.766117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:05.766117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:05.766117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:05.766117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:05.766117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:05.766940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:05.766940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:05.766940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:05.766940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:05.766940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:05.766940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:05.766940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:05.766940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:05.766940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:05.766940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:05.766940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:05.766940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:05.766940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:05.766940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:05.766940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:05.766940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:05.766940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:05.767761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:05.767761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:05.767761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:05.767761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:05.767761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:05.767761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:05.767761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:05.767761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:05.770775 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 269/538] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:05.776355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadf4e60-df13-4262-a2ca-f01469528f61 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:05.872704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:05.891525 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadf4e60-df13-4262-a2ca-f01469528f61 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:05.898463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:05.933138 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1766d4-1571-4bac-a1c2-1320504ca5ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:05.934728 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1766d4-1571-4bac-a1c2-1320504ca5ac None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:05.935193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1766d4-1571-4bac-a1c2-1320504ca5ac None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:05.935639 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1766d4-1571-4bac-a1c2-1320504ca5ac None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:05.936634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1766d4-1571-4bac-a1c2-1320504ca5ac None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:05.937413 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1766d4-1571-4bac-a1c2-1320504ca5ac None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:05.939359 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1766d4-1571-4bac-a1c2-1320504ca5ac None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:05.976793 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 270/539] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:05.998059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1766d4-1571-4bac-a1c2-1320504ca5ac None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:06.016531 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0586fb44-8b7a-4b2b-b1f7-4a0ac1a25709 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:06.053831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bee0968-5143-4696-84f2-245e97ef49a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:06.054761 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bee0968-5143-4696-84f2-245e97ef49a4 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:06.055276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bee0968-5143-4696-84f2-245e97ef49a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:06.055720 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bee0968-5143-4696-84f2-245e97ef49a4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:06.056644 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bee0968-5143-4696-84f2-245e97ef49a4 None admin] RBAC: Adding query filter params (role.id=df912a77fbaa479db172f76a214f8bc8, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=da4f44b86ca94adf959d9a41d915c902) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:06.057156 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bee0968-5143-4696-84f2-245e97ef49a4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:06.058751 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bee0968-5143-4696-84f2-245e97ef49a4 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:06.065108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 270/540] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:05 2022] GET /identity/v3/role_assignments?user.id=da4f44b86ca94adf959d9a41d915c902&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=df912a77fbaa479db172f76a214f8bc8 => generated 259 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:06.115706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1766d4-1571-4bac-a1c2-1320504ca5ac None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:06.160719 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52020d64-a197-4ca7-a9a4-c6783af6fa00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:06.161395 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52020d64-a197-4ca7-a9a4-c6783af6fa00 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:06.161675 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52020d64-a197-4ca7-a9a4-c6783af6fa00 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:06.161943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52020d64-a197-4ca7-a9a4-c6783af6fa00 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:06.162785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52020d64-a197-4ca7-a9a4-c6783af6fa00 None admin] RBAC: Adding query filter params (role.id=df912a77fbaa479db172f76a214f8bc8, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=615a607b6be14b82979f94eabaf4c34c) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:06.163098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52020d64-a197-4ca7-a9a4-c6783af6fa00 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:06.164583 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52020d64-a197-4ca7-a9a4-c6783af6fa00 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:06.171041 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 271/541] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:05 2022] GET /identity/v3/role_assignments?user.id=615a607b6be14b82979f94eabaf4c34c&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=df912a77fbaa479db172f76a214f8bc8 => generated 259 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:06.439241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf87166-687a-4057-8541-dc3d154699f1 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:06.439754 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf87166-687a-4057-8541-dc3d154699f1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:06.439859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf87166-687a-4057-8541-dc3d154699f1 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:06.441670 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 271/542] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:47:06.447388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ae7ce1-0072-44be-9b84-cfefd986d9ee None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:06.447816 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ae7ce1-0072-44be-9b84-cfefd986d9ee None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:06.447990 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ae7ce1-0072-44be-9b84-cfefd986d9ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:06.474050 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-56ae7ce1-0072-44be-9b84-cfefd986d9ee None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:06.475277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-56ae7ce1-0072-44be-9b84-cfefd986d9ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:06.477478 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:06.578982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 272/543] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:06.875718 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf87166-687a-4057-8541-dc3d154699f1 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:07.002736 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf87166-687a-4057-8541-dc3d154699f1 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:07.027553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:07.046779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8c91e0-2ec5-47e6-9161-e2e2fa906f97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:07.047408 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8c91e0-2ec5-47e6-9161-e2e2fa906f97 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:07.047686 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8c91e0-2ec5-47e6-9161-e2e2fa906f97 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:07.047985 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8c91e0-2ec5-47e6-9161-e2e2fa906f97 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:07.061349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8c91e0-2ec5-47e6-9161-e2e2fa906f97 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:07.066090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-9d8c91e0-2ec5-47e6-9161-e2e2fa906f97 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:07.066090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:07.066090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:07.066090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:07.066090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:07.066090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:07.066090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:07.066090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:07.066090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:07.066090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:07.066090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:07.066090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:07.066090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:07.066090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:07.066090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:07.066090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:07.066989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:07.066989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:07.066989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:07.066989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:07.066989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:07.066989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:07.066989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:07.066989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:07.066989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:07.066989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:07.066989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:07.066989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:07.066989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:07.066989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:07.066989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:07.066989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:07.067767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:07.067767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:07.067767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:07.067767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:07.067767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:07.067767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:07.067767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:07.067767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:07.067767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:07.067767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:07.067767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:07.067767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:07.067767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 272/544] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:07.073156 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ae7ce1-0072-44be-9b84-cfefd986d9ee None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:07.241210 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ae7ce1-0072-44be-9b84-cfefd986d9ee None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:07.288140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cc9586-cc42-488a-b797-0f546bcffdbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:07.288754 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cc9586-cc42-488a-b797-0f546bcffdbe None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:07.289029 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cc9586-cc42-488a-b797-0f546bcffdbe None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:07.289324 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cc9586-cc42-488a-b797-0f546bcffdbe None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:07.290001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cc9586-cc42-488a-b797-0f546bcffdbe None admin] RBAC: Adding query filter params (name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:07.290306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cc9586-cc42-488a-b797-0f546bcffdbe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:07.291687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cc9586-cc42-488a-b797-0f546bcffdbe None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:07.301617 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:07.305177 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 273/545] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:07.310469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8c91e0-2ec5-47e6-9161-e2e2fa906f97 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:07.449438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8c91e0-2ec5-47e6-9161-e2e2fa906f97 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:07.525950 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea6f66f-e992-4750-820a-65d81be9de6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:07.526533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea6f66f-e992-4750-820a-65d81be9de6e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:07.526810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea6f66f-e992-4750-820a-65d81be9de6e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:07.527077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea6f66f-e992-4750-820a-65d81be9de6e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:07.545590 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea6f66f-e992-4750-820a-65d81be9de6e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:07.547056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea6f66f-e992-4750-820a-65d81be9de6e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:07.569889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-cea6f66f-e992-4750-820a-65d81be9de6e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:07.569889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:07.569889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:07.569889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:07.569889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:07.569889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:07.569889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:07.569889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:07.569889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:07.569889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:07.569889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:07.569889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:07.569889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:07.569889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:07.569889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:07.569889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:07.571321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:07.571321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:07.571321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:07.571321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:07.571321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:07.571321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:07.571321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:07.571321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:07.571321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:07.571321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:07.571321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:07.571321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:07.571321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:07.571321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:07.571321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:07.571321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:07.571321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:07.572924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:07.572924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:07.572924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:07.572924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 273/546] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:07.580131 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cc9586-cc42-488a-b797-0f546bcffdbe None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:07.733626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cc9586-cc42-488a-b797-0f546bcffdbe None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:07.736149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b99dd5-546f-488c-a249-89c3d61c55ec None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:07.741099 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b99dd5-546f-488c-a249-89c3d61c55ec None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:07.741099 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b99dd5-546f-488c-a249-89c3d61c55ec None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:07.741287 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 274/547] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:47:07.752659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d17890-085a-4fa0-9951-30b5d3f9d3d7 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:07.753262 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d17890-085a-4fa0-9951-30b5d3f9d3d7 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:07.753418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d17890-085a-4fa0-9951-30b5d3f9d3d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:07.771071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cefe48-ef95-43a3-8793-d62a9e4fe3db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:07.771676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cefe48-ef95-43a3-8793-d62a9e4fe3db None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:07.771952 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cefe48-ef95-43a3-8793-d62a9e4fe3db None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:07.772223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cefe48-ef95-43a3-8793-d62a9e4fe3db None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:07.772864 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cefe48-ef95-43a3-8793-d62a9e4fe3db None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:07.773192 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cefe48-ef95-43a3-8793-d62a9e4fe3db None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:07.774549 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cefe48-ef95-43a3-8793-d62a9e4fe3db None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:07.787502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 274/548] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:07.790084 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-f0d17890-085a-4fa0-9951-30b5d3f9d3d7 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:07.791272 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f0d17890-085a-4fa0-9951-30b5d3f9d3d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:07.794124 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cefe48-ef95-43a3-8793-d62a9e4fe3db None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:07.856928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:07.857905 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:07.897150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cefe48-ef95-43a3-8793-d62a9e4fe3db None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:07.931773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 275/549] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:07.934306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5fbcd8-f6ad-432d-91c3-a18824b4dfb2 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:07.934593 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5fbcd8-f6ad-432d-91c3-a18824b4dfb2 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:07.934870 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5fbcd8-f6ad-432d-91c3-a18824b4dfb2 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:07.936886 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 276/550] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:07 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:07.943551 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae434cb-a31d-4262-91c2-377ac58d6bf5 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:07.943815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae434cb-a31d-4262-91c2-377ac58d6bf5 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:07.944097 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae434cb-a31d-4262-91c2-377ac58d6bf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:07.953379 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df93ff6-26a8-4fb2-b369-37bccc9756ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:07.954338 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df93ff6-26a8-4fb2-b369-37bccc9756ab None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:07.954819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df93ff6-26a8-4fb2-b369-37bccc9756ab None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:07.955308 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df93ff6-26a8-4fb2-b369-37bccc9756ab None admin] PATH_INFO: `/v3/users/swift` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:07.964973 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-fae434cb-a31d-4262-91c2-377ac58d6bf5 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:07.966291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-fae434cb-a31d-4262-91c2-377ac58d6bf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:07.969455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df93ff6-26a8-4fb2-b369-37bccc9756ab None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:07.971108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df93ff6-26a8-4fb2-b369-37bccc9756ab None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:08.006360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-0df93ff6-26a8-4fb2-b369-37bccc9756ab None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 04 00:47:08.006360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:08.006360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:08.006360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:08.006360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:08.006360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:08.006360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:08.006360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:08.006360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:08.006360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:08.006360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:08.006360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:08.006360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:08.006360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:08.006360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:08.006360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:08.007388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:08.008272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:08.008272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:08.008272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:08.008272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:08.008272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:08.008272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:08.008272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:08.008272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:08.008272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 04 00:47:08.008272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:08.017300 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 275/551] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:47:07 2022] GET /identity/v3/users/swift => generated 83 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:08.023237 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df93ff6-26a8-4fb2-b369-37bccc9756ab None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:08.151323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 277/552] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:08.154114 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df93ff6-26a8-4fb2-b369-37bccc9756ab None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:08.181480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:08.189649 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318ebe07-d258-4a74-a01f-76c11598077d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:08.191120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318ebe07-d258-4a74-a01f-76c11598077d None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:08.191603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318ebe07-d258-4a74-a01f-76c11598077d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:08.192040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318ebe07-d258-4a74-a01f-76c11598077d None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:08.192934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318ebe07-d258-4a74-a01f-76c11598077d None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:08.193430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318ebe07-d258-4a74-a01f-76c11598077d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:08.196044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318ebe07-d258-4a74-a01f-76c11598077d None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:08.217307 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae434cb-a31d-4262-91c2-377ac58d6bf5 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:08.235014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 276/553] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Sep 4 00:47:08 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:08.242714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318ebe07-d258-4a74-a01f-76c11598077d None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:08.356958 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:08.373101 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae434cb-a31d-4262-91c2-377ac58d6bf5 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:08.408699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318ebe07-d258-4a74-a01f-76c11598077d None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:08.485392 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228dd0c8-3092-4ada-89ec-87cd5b4b2fd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:08.486073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228dd0c8-3092-4ada-89ec-87cd5b4b2fd3 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:08.486315 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228dd0c8-3092-4ada-89ec-87cd5b4b2fd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:08.486588 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228dd0c8-3092-4ada-89ec-87cd5b4b2fd3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:08.496447 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838a9f3c-29cc-4ab1-ba73-de2eac7a0032 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:08.497064 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a9f3c-29cc-4ab1-ba73-de2eac7a0032 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:08.497398 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a9f3c-29cc-4ab1-ba73-de2eac7a0032 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:08.503071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a9f3c-29cc-4ab1-ba73-de2eac7a0032 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:08.503180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228dd0c8-3092-4ada-89ec-87cd5b4b2fd3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:08.504073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-228dd0c8-3092-4ada-89ec-87cd5b4b2fd3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:08.504073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:08.504073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:08.504073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:08.504073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:08.504073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:08.504073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:08.504073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:08.504073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:08.504073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:08.504073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:08.504073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:08.504073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:08.504073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:08.504073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:08.504073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:08.505013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:08.505013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:08.505013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:08.505013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:08.505013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:08.505013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:08.505013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:08.505013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:08.505013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:08.505013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:08.505013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:08.505013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:08.505013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:08.505013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:08.505013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:08.505013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:08.506599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:08.506599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:08.506599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:08.506599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:08.506599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:08.506599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:08.506599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:08.506599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:08.506599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:08.506599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:08.506599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:08.506599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:08.506599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 278/554] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:08.506599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228dd0c8-3092-4ada-89ec-87cd5b4b2fd3 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:08.553196 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:08.577150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838a9f3c-29cc-4ab1-ba73-de2eac7a0032 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:08.578633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838a9f3c-29cc-4ab1-ba73-de2eac7a0032 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:08.626543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228dd0c8-3092-4ada-89ec-87cd5b4b2fd3 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:08.663575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aaf4cb-90dd-4d06-bfe2-e7ca27080c2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:08.664113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aaf4cb-90dd-4d06-bfe2-e7ca27080c2a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:08.664395 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aaf4cb-90dd-4d06-bfe2-e7ca27080c2a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:08.664668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aaf4cb-90dd-4d06-bfe2-e7ca27080c2a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:08.674780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2aaf4cb-90dd-4d06-bfe2-e7ca27080c2a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:08.679827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-a2aaf4cb-90dd-4d06-bfe2-e7ca27080c2a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:08.679827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:08.679827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:08.679827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:08.679827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:08.679827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:08.679827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:08.679827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:08.679827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:08.679827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:08.679827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:08.679827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:08.679827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:08.679827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:08.679827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:08.679827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:08.680771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:08.680771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:08.680771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:08.680771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:08.680771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:08.680771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:08.680771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:08.680771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:08.680771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:08.680771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:08.680771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:08.680771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:08.680771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:08.680771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:08.680771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:08.680771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:08.681702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:08.681702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:08.681702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:08.681702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:08.681702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:08.681702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:08.681702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:08.681702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:08.681702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:08.681702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:08.681702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:08.681702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:08.681702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 279/555] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:08.682370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aaf4cb-90dd-4d06-bfe2-e7ca27080c2a None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:08.689821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 277/556] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 447 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:47:08.692288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1339c7a9-48c5-4efc-a673-58d3eab5abb3 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:08.692577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1339c7a9-48c5-4efc-a673-58d3eab5abb3 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:08.692852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1339c7a9-48c5-4efc-a673-58d3eab5abb3 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:08.694225 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 278/557] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:08.695448 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccff820-b7b9-4397-9700-8ecfabe7e4ce None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:08.695734 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccff820-b7b9-4397-9700-8ecfabe7e4ce None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:08.696005 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccff820-b7b9-4397-9700-8ecfabe7e4ce None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:08.697211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 279/558] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:08 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:08.697896 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccff820-b7b9-4397-9700-8ecfabe7e4ce None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:08.701451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:08.777022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:08.787717 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aaf4cb-90dd-4d06-bfe2-e7ca27080c2a None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:08.798329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccff820-b7b9-4397-9700-8ecfabe7e4ce None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:08.821450 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5aa6f4-23f1-41fb-aaf2-1a57196cbf3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:08.822062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5aa6f4-23f1-41fb-aaf2-1a57196cbf3f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:08.822338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5aa6f4-23f1-41fb-aaf2-1a57196cbf3f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:08.822613 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5aa6f4-23f1-41fb-aaf2-1a57196cbf3f None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:08.823283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5aa6f4-23f1-41fb-aaf2-1a57196cbf3f None admin] RBAC: Adding query filter params (name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:08.823586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5aa6f4-23f1-41fb-aaf2-1a57196cbf3f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:08.824929 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5aa6f4-23f1-41fb-aaf2-1a57196cbf3f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:08.833248 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24704624-0c3b-49d9-9e4f-01989a26912f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:08.833765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 280/559] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:08.833881 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24704624-0c3b-49d9-9e4f-01989a26912f None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:08.834150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24704624-0c3b-49d9-9e4f-01989a26912f None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:08.834430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24704624-0c3b-49d9-9e4f-01989a26912f None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:08.834595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5aa6f4-23f1-41fb-aaf2-1a57196cbf3f None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:08.835060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24704624-0c3b-49d9-9e4f-01989a26912f None admin] RBAC: Adding query filter params (name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:08.835359 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24704624-0c3b-49d9-9e4f-01989a26912f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:08.836705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24704624-0c3b-49d9-9e4f-01989a26912f None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:08.845532 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 280/560] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:08.846700 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da30b15-69c0-4a1b-bfc4-8bfb1e585a00 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:08.846986 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da30b15-69c0-4a1b-bfc4-8bfb1e585a00 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:08.847255 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da30b15-69c0-4a1b-bfc4-8bfb1e585a00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:08.864534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-0da30b15-69c0-4a1b-bfc4-8bfb1e585a00 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:08.865865 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0da30b15-69c0-4a1b-bfc4-8bfb1e585a00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:08.919107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:08.933410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 281/561] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:08.934940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adf9ee1-e87b-486e-be87-efbad450a1a1 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:08.935428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adf9ee1-e87b-486e-be87-efbad450a1a1 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:08.935847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adf9ee1-e87b-486e-be87-efbad450a1a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:08.943301 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5aa6f4-23f1-41fb-aaf2-1a57196cbf3f None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:08.961959 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-8adf9ee1-e87b-486e-be87-efbad450a1a1 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:08.963144 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-8adf9ee1-e87b-486e-be87-efbad450a1a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:08.978394 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23539e3d-ce49-4e31-9e3e-ed4dc26e4b06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:08.979016 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23539e3d-ce49-4e31-9e3e-ed4dc26e4b06 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:08.979292 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23539e3d-ce49-4e31-9e3e-ed4dc26e4b06 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:08.979562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23539e3d-ce49-4e31-9e3e-ed4dc26e4b06 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:08.993117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23539e3d-ce49-4e31-9e3e-ed4dc26e4b06 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:08.995258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23539e3d-ce49-4e31-9e3e-ed4dc26e4b06 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:09.014715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-23539e3d-ce49-4e31-9e3e-ed4dc26e4b06 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:09.014715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:09.014715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:09.014715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:09.014715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:09.014715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:09.014715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:09.014715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:09.014715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:09.014715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:09.014715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:09.014715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:09.014715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:09.014715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:09.014715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:09.014715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:09.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:09.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:09.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:09.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:09.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:09.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:09.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:09.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:09.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:09.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:09.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:09.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:09.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:09.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:09.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:09.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:09.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:09.016599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:09.016599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:09.016599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:09.020934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 281/562] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:09.022017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23539e3d-ce49-4e31-9e3e-ed4dc26e4b06 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:09.031946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:09.037810 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 282/563] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:09.038834 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adf9ee1-e87b-486e-be87-efbad450a1a1 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:09.166070 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23539e3d-ce49-4e31-9e3e-ed4dc26e4b06 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:09.168600 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adf9ee1-e87b-486e-be87-efbad450a1a1 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:09.202169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:09.206923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70d12d0-ba1b-41f6-b040-c7df1b899e19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:09.207794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d12d0-ba1b-41f6-b040-c7df1b899e19 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:09.208073 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d12d0-ba1b-41f6-b040-c7df1b899e19 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:09.208482 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d12d0-ba1b-41f6-b040-c7df1b899e19 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:09.211176 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f27a81-fdb6-4d0d-ac06-a0be948f5d52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:09.218319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f27a81-fdb6-4d0d-ac06-a0be948f5d52 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:09.218319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f27a81-fdb6-4d0d-ac06-a0be948f5d52 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:09.218319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f27a81-fdb6-4d0d-ac06-a0be948f5d52 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:09.218764 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70d12d0-ba1b-41f6-b040-c7df1b899e19 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:09.220186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70d12d0-ba1b-41f6-b040-c7df1b899e19 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:09.222523 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f27a81-fdb6-4d0d-ac06-a0be948f5d52 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:09.223919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f27a81-fdb6-4d0d-ac06-a0be948f5d52 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:09.235027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-e70d12d0-ba1b-41f6-b040-c7df1b899e19 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:09.235027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:09.235027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:09.235027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:09.235027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:09.235027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:09.235027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:09.235027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:09.235027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:09.235027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:09.235027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:09.235027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:09.235027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:09.235027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:09.235027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:09.235027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:09.235939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:09.235939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:09.235939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:09.235939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:09.235939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:09.235939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:09.235939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:09.235939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:09.235939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:09.235939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:09.235939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:09.235939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:09.235939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:09.235939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:09.235939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:09.235939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:09.235939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:09.242419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:09.242419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:09.242419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:09.242419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 283/564] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:09.242419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70d12d0-ba1b-41f6-b040-c7df1b899e19 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:09.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-a0f27a81-fdb6-4d0d-ac06-a0be948f5d52 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:09.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:09.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:09.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:09.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:09.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:09.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:09.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:09.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:09.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:09.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:09.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:09.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:09.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:09.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:09.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:09.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:09.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:09.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:09.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:09.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:09.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:09.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:09.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:09.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:09.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:09.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:09.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:09.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:09.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:09.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:09.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:09.243510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:09.244378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:09.244378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:09.244378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:09.244378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 282/565] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:09.244378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f27a81-fdb6-4d0d-ac06-a0be948f5d52 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:09.297726 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:09.347320 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70d12d0-ba1b-41f6-b040-c7df1b899e19 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:09.348793 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f27a81-fdb6-4d0d-ac06-a0be948f5d52 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:09.380570 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b3327d-78fa-4692-aa74-3d18dcb3faa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:09.381194 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b3327d-78fa-4692-aa74-3d18dcb3faa2 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:09.381477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b3327d-78fa-4692-aa74-3d18dcb3faa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:09.381741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b3327d-78fa-4692-aa74-3d18dcb3faa2 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:09.382378 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b3327d-78fa-4692-aa74-3d18dcb3faa2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:09.382678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b3327d-78fa-4692-aa74-3d18dcb3faa2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:09.382838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1c5caa-71f1-465e-b255-4f0dff317386 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:09.383496 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1c5caa-71f1-465e-b255-4f0dff317386 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:09.383775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1c5caa-71f1-465e-b255-4f0dff317386 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:09.384035 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b3327d-78fa-4692-aa74-3d18dcb3faa2 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:09.384130 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1c5caa-71f1-465e-b255-4f0dff317386 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:09.394211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1c5caa-71f1-465e-b255-4f0dff317386 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:09.395706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 284/566] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:09.396495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b3327d-78fa-4692-aa74-3d18dcb3faa2 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:09.399586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-8e1c5caa-71f1-465e-b255-4f0dff317386 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:09.399586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:09.399586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:09.399586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:09.399586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:09.399586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:09.399586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:09.399586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:09.399586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:09.399586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:09.399586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:09.399586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:09.399586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:09.399586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:09.399586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:09.399586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:09.400543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:09.400543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:09.400543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:09.400543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:09.400543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:09.400543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:09.400543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:09.400543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:09.400543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:09.400543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:09.400543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:09.400543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:09.400543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:09.400543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:09.400543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:09.400543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:09.401405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:09.401405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:09.401405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:09.401405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:09.401405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:09.401405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:09.401405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:09.401405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:09.401405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:09.401405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:09.401405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:09.401405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:09.401405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 283/567] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:09.402079 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1c5caa-71f1-465e-b255-4f0dff317386 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:09.440986 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:09.502612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1c5caa-71f1-465e-b255-4f0dff317386 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:09.503238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b3327d-78fa-4692-aa74-3d18dcb3faa2 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:09.515823 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:09.535765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1aabb1f-e13c-46e8-b566-ccc9b0571d5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:09.536362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1aabb1f-e13c-46e8-b566-ccc9b0571d5e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:09.536491 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16161bbf-ff77-4b69-9b43-d15b22bf02d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:09.536638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1aabb1f-e13c-46e8-b566-ccc9b0571d5e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:09.536909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1aabb1f-e13c-46e8-b566-ccc9b0571d5e None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:09.537104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16161bbf-ff77-4b69-9b43-d15b22bf02d6 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:09.537386 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16161bbf-ff77-4b69-9b43-d15b22bf02d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:09.537579 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1aabb1f-e13c-46e8-b566-ccc9b0571d5e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:09.537669 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16161bbf-ff77-4b69-9b43-d15b22bf02d6 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:09.537881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1aabb1f-e13c-46e8-b566-ccc9b0571d5e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:09.538289 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16161bbf-ff77-4b69-9b43-d15b22bf02d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:09.538596 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16161bbf-ff77-4b69-9b43-d15b22bf02d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:09.539231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1aabb1f-e13c-46e8-b566-ccc9b0571d5e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:09.539947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16161bbf-ff77-4b69-9b43-d15b22bf02d6 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:09.551140 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 284/568] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:09.551929 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1aabb1f-e13c-46e8-b566-ccc9b0571d5e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:09.552126 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 285/569] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:09.552934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16161bbf-ff77-4b69-9b43-d15b22bf02d6 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:09.648239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:09.650529 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16161bbf-ff77-4b69-9b43-d15b22bf02d6 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:09.652073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1aabb1f-e13c-46e8-b566-ccc9b0571d5e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:09.683865 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5743a62a-1585-46fa-bf71-cd53d80c1b78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:09.684496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5743a62a-1585-46fa-bf71-cd53d80c1b78 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:09.684808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5743a62a-1585-46fa-bf71-cd53d80c1b78 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:09.685112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5743a62a-1585-46fa-bf71-cd53d80c1b78 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:09.686162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9680d04-d16a-4f68-b6cc-b8b2e690957b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:09.686775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9680d04-d16a-4f68-b6cc-b8b2e690957b None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:09.687061 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9680d04-d16a-4f68-b6cc-b8b2e690957b None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:09.687330 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9680d04-d16a-4f68-b6cc-b8b2e690957b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:09.692391 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5743a62a-1585-46fa-bf71-cd53d80c1b78 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:09.693870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5743a62a-1585-46fa-bf71-cd53d80c1b78 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:09.697911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9680d04-d16a-4f68-b6cc-b8b2e690957b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:09.702648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-a9680d04-d16a-4f68-b6cc-b8b2e690957b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:09.702648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:09.702648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:09.702648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:09.702648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:09.702648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:09.702648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:09.702648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:09.702648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:09.702648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:09.702648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:09.702648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:09.702648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:09.702648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:09.702648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:09.702648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:09.703719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:09.703719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:09.703719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:09.703719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:09.703719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:09.703719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:09.703719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:09.703719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:09.703719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:09.703719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:09.703719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:09.703719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:09.703719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:09.703719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:09.703719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:09.703719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:09.704558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:09.704558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:09.704558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:09.704558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:09.704558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:09.704558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:09.704558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:09.704558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:09.704558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:09.704558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:09.704558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:09.704558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:09.704558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 285/570] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:09.705259 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9680d04-d16a-4f68-b6cc-b8b2e690957b None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:09.705327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-5743a62a-1585-46fa-bf71-cd53d80c1b78 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:09.705327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:09.705327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:09.705327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:09.705327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:09.705327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:09.705327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:09.705327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:09.705327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:09.705327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:09.705327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:09.705327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:09.705327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:09.705327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:09.705327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:09.706714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:09.706714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:09.706714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:09.706714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:09.706714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:09.706714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:09.706714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:09.706714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:09.706714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:09.706714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:09.706714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:09.706714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:09.706714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:09.706714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:09.706714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:09.706714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:09.706714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:09.707571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:09.707571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:09.707571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:09.707571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:09.707571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:09.707571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:09.707571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:09.707571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:09.707571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 286/571] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:09.707571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5743a62a-1585-46fa-bf71-cd53d80c1b78 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:09.738412 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:09.805653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5743a62a-1585-46fa-bf71-cd53d80c1b78 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:09.807241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9680d04-d16a-4f68-b6cc-b8b2e690957b None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:09.839969 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ec8e30-9486-4241-b7fd-01da4b8e8046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:09.840340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea3abec-6dc3-4fbb-b80a-ed934b047f99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:09.840585 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec8e30-9486-4241-b7fd-01da4b8e8046 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:09.840862 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec8e30-9486-4241-b7fd-01da4b8e8046 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:09.840960 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea3abec-6dc3-4fbb-b80a-ed934b047f99 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:09.841156 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec8e30-9486-4241-b7fd-01da4b8e8046 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:09.841341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea3abec-6dc3-4fbb-b80a-ed934b047f99 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:09.841482 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea3abec-6dc3-4fbb-b80a-ed934b047f99 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:09.842138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea3abec-6dc3-4fbb-b80a-ed934b047f99 None admin] RBAC: Adding query filter params (name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:09.842438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea3abec-6dc3-4fbb-b80a-ed934b047f99 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:09.843795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea3abec-6dc3-4fbb-b80a-ed934b047f99 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:09.848180 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ec8e30-9486-4241-b7fd-01da4b8e8046 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:09.849638 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ec8e30-9486-4241-b7fd-01da4b8e8046 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:09.851528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 286/572] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:09.852308 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea3abec-6dc3-4fbb-b80a-ed934b047f99 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:09.866042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-d4ec8e30-9486-4241-b7fd-01da4b8e8046 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 04 00:47:09.866042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:09.866042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:09.866042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:09.866042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:09.866042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:09.866042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:09.866042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:09.866042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:09.866042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:09.866042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:09.866042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:09.866042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:09.866042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:09.866042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:09.866042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:09.867061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:09.868024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:09.868024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:09.868024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:09.868024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:09.868024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:09.868024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:09.868024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:09.868024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:09.868024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 04 00:47:09.868024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:09.868024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 287/573] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:47:09 2022] GET /identity/v3/users/glance => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:09.868024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ec8e30-9486-4241-b7fd-01da4b8e8046 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:09.878879 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:09.958216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea3abec-6dc3-4fbb-b80a-ed934b047f99 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:09.959827 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:09.968873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ec8e30-9486-4241-b7fd-01da4b8e8046 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:09.992117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0094ce3-0855-44d7-bda7-be688b345f04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:09.992665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0094ce3-0855-44d7-bda7-be688b345f04 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:09.992938 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0094ce3-0855-44d7-bda7-be688b345f04 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:09.993248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0094ce3-0855-44d7-bda7-be688b345f04 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:10.000253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0094ce3-0855-44d7-bda7-be688b345f04 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:10.001709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0094ce3-0855-44d7-bda7-be688b345f04 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:10.002763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb13329f-c7c0-4a5d-9ff0-51777f8a67e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:10.004064 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb13329f-c7c0-4a5d-9ff0-51777f8a67e9 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:10.004436 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb13329f-c7c0-4a5d-9ff0-51777f8a67e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:10.004718 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb13329f-c7c0-4a5d-9ff0-51777f8a67e9 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:10.005512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb13329f-c7c0-4a5d-9ff0-51777f8a67e9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:10.005816 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb13329f-c7c0-4a5d-9ff0-51777f8a67e9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:10.007166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb13329f-c7c0-4a5d-9ff0-51777f8a67e9 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:10.013690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-a0094ce3-0855-44d7-bda7-be688b345f04 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:47:10.013690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:10.013690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:10.013690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:10.013690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:10.013690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:10.013690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:10.013690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:10.013690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:10.013690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:10.013690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:10.013690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:10.013690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:10.013690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:10.013690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:10.013690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:10.020310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:10.021418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:10.021418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:10.021418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:10.021418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:10.021418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:10.021418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:10.021418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:10.021418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:10.021418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:47:10.021418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:10.021418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 287/574] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 4 00:47:09 2022] GET /identity/v3/users/nova => generated 82 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:10.022018 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 288/575] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:10.022018 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb13329f-c7c0-4a5d-9ff0-51777f8a67e9 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:10.022153 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0094ce3-0855-44d7-bda7-be688b345f04 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:10.100064 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:10.122165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb13329f-c7c0-4a5d-9ff0-51777f8a67e9 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:10.125412 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0094ce3-0855-44d7-bda7-be688b345f04 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:10.156600 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbc1af0-4836-409f-9151-9d54b3f5f4c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:10.157247 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbc1af0-4836-409f-9151-9d54b3f5f4c2 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:10.157524 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbc1af0-4836-409f-9151-9d54b3f5f4c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:10.157792 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbc1af0-4836-409f-9151-9d54b3f5f4c2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:10.159920 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a224bcc-df41-4938-a1a6-de2f758e970d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:10.160531 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a224bcc-df41-4938-a1a6-de2f758e970d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:10.160821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a224bcc-df41-4938-a1a6-de2f758e970d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:10.161117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a224bcc-df41-4938-a1a6-de2f758e970d None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:10.161827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a224bcc-df41-4938-a1a6-de2f758e970d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:10.162137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a224bcc-df41-4938-a1a6-de2f758e970d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:10.163562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a224bcc-df41-4938-a1a6-de2f758e970d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:10.168325 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbc1af0-4836-409f-9151-9d54b3f5f4c2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:10.169768 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbc1af0-4836-409f-9151-9d54b3f5f4c2 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:10.175577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 288/576] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:10.176371 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a224bcc-df41-4938-a1a6-de2f758e970d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:10.188634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-5fbc1af0-4836-409f-9151-9d54b3f5f4c2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:10.188634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:10.188634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:10.188634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:10.188634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:10.188634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:10.188634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:10.188634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:10.188634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:10.188634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:10.188634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:10.188634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:10.188634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:10.188634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:10.188634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:10.188634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:10.189797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:10.189797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:10.189797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:10.189797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:10.189797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:10.189797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:10.189797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:10.189797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:10.189797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:10.189797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:10.189797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:10.189797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:10.189797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:10.189797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:10.189797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:10.189797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:10.189797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:10.191205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:10.191205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:10.191205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:10.193283 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 289/577] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:10.194382 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbc1af0-4836-409f-9151-9d54b3f5f4c2 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:10.203033 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:10.277688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a224bcc-df41-4938-a1a6-de2f758e970d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:10.292607 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbc1af0-4836-409f-9151-9d54b3f5f4c2 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:10.310830 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3eaec1-188e-43e6-9387-a9b6822c8191 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:10.311427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3eaec1-188e-43e6-9387-a9b6822c8191 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:10.311708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3eaec1-188e-43e6-9387-a9b6822c8191 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:10.311976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3eaec1-188e-43e6-9387-a9b6822c8191 None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:10.312696 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3eaec1-188e-43e6-9387-a9b6822c8191 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:10.313004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3eaec1-188e-43e6-9387-a9b6822c8191 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:10.314435 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3eaec1-188e-43e6-9387-a9b6822c8191 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:10.323635 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:10.326204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377f5734-9bf2-44c9-9d63-f60025d97275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:10.326806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377f5734-9bf2-44c9-9d63-f60025d97275 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:10.327092 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377f5734-9bf2-44c9-9d63-f60025d97275 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:10.327365 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377f5734-9bf2-44c9-9d63-f60025d97275 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:10.328127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377f5734-9bf2-44c9-9d63-f60025d97275 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:10.328400 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377f5734-9bf2-44c9-9d63-f60025d97275 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:10.329845 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377f5734-9bf2-44c9-9d63-f60025d97275 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:10.333817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 289/578] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Sep 4 00:47:10 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:10.334132 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3eaec1-188e-43e6-9387-a9b6822c8191 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:10.352147 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 290/579] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 4 00:47:10 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:10.353210 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377f5734-9bf2-44c9-9d63-f60025d97275 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:10.432705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3eaec1-188e-43e6-9387-a9b6822c8191 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:10.441955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:10.468198 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5fc2e3-32a7-471b-914a-eb85255618c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:10.468863 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5fc2e3-32a7-471b-914a-eb85255618c6 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:10.469172 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5fc2e3-32a7-471b-914a-eb85255618c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:10.469453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5fc2e3-32a7-471b-914a-eb85255618c6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:10.479293 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377f5734-9bf2-44c9-9d63-f60025d97275 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:10.479529 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5fc2e3-32a7-471b-914a-eb85255618c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:10.480964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5fc2e3-32a7-471b-914a-eb85255618c6 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:10.495859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-de5fc2e3-32a7-471b-914a-eb85255618c6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:10.495859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:10.495859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:10.495859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:10.495859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:10.495859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:10.495859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:10.495859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:10.495859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:10.495859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:10.495859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:10.495859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:10.495859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:10.495859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:10.495859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:10.495859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:10.496872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:10.496872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:10.496872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:10.496872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:10.496872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:10.496872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:10.496872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:10.496872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:10.496872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:10.496872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:10.496872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:10.496872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:10.496872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:10.496872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:10.496872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:10.496872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:10.496872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:10.498147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:10.498147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:10.498147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:10.498147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 290/580] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:10.498147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5fc2e3-32a7-471b-914a-eb85255618c6 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:10.519544 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722643cf-bd24-4322-8e1c-28af334a7670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:10.520149 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722643cf-bd24-4322-8e1c-28af334a7670 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:10.520432 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722643cf-bd24-4322-8e1c-28af334a7670 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:10.520805 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722643cf-bd24-4322-8e1c-28af334a7670 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:10.521573 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722643cf-bd24-4322-8e1c-28af334a7670 None admin] RBAC: Adding query filter params (role.id=df912a77fbaa479db172f76a214f8bc8, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=123ab6ff796c4e599ce1d6f4095bd898) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:10.521888 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722643cf-bd24-4322-8e1c-28af334a7670 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:10.523324 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722643cf-bd24-4322-8e1c-28af334a7670 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:10.528895 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 291/581] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:10 2022] GET /identity/v3/role_assignments?user.id=123ab6ff796c4e599ce1d6f4095bd898&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=df912a77fbaa479db172f76a214f8bc8 => generated 259 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:10.529662 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722643cf-bd24-4322-8e1c-28af334a7670 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:10.550836 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:10.609995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5fc2e3-32a7-471b-914a-eb85255618c6 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:10.647768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c305712a-2fbb-48fa-8d00-ba4d64183f4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:10.648388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c305712a-2fbb-48fa-8d00-ba4d64183f4a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:10.648669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c305712a-2fbb-48fa-8d00-ba4d64183f4a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:10.648944 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c305712a-2fbb-48fa-8d00-ba4d64183f4a None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:10.649349 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722643cf-bd24-4322-8e1c-28af334a7670 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:10.649654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c305712a-2fbb-48fa-8d00-ba4d64183f4a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:10.649952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c305712a-2fbb-48fa-8d00-ba4d64183f4a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:10.651340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c305712a-2fbb-48fa-8d00-ba4d64183f4a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:10.664569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 291/582] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:10.664924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c305712a-2fbb-48fa-8d00-ba4d64183f4a None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:10.678123 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:10.687426 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fa1b39-a914-4481-ba52-d79fc0f94ec7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:10.688054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fa1b39-a914-4481-ba52-d79fc0f94ec7 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:10.689174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fa1b39-a914-4481-ba52-d79fc0f94ec7 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:10.689174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fa1b39-a914-4481-ba52-d79fc0f94ec7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:10.703154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fa1b39-a914-4481-ba52-d79fc0f94ec7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:10.704620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fa1b39-a914-4481-ba52-d79fc0f94ec7 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:10.723735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-50fa1b39-a914-4481-ba52-d79fc0f94ec7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:10.723735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:10.723735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:10.723735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:10.723735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:10.723735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:10.723735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:10.723735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:10.723735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:10.723735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:10.723735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:10.723735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:10.723735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:10.723735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:10.723735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:10.723735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:10.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:10.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:10.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:10.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:10.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:10.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:10.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:10.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:10.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:10.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:10.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:10.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:10.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:10.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:10.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:10.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:10.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:10.725609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:10.725609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:10.725609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:10.725609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 292/583] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:10.725847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fa1b39-a914-4481-ba52-d79fc0f94ec7 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:10.781747 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c305712a-2fbb-48fa-8d00-ba4d64183f4a None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:10.796540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:10.822161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49275aed-1680-4a9b-bdfe-f32a154f207a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:10.822908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49275aed-1680-4a9b-bdfe-f32a154f207a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:10.823187 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49275aed-1680-4a9b-bdfe-f32a154f207a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:10.823666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49275aed-1680-4a9b-bdfe-f32a154f207a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:10.829402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fa1b39-a914-4481-ba52-d79fc0f94ec7 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:10.835823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49275aed-1680-4a9b-bdfe-f32a154f207a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:10.837395 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49275aed-1680-4a9b-bdfe-f32a154f207a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:10.853271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-49275aed-1680-4a9b-bdfe-f32a154f207a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:10.853271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:10.853271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:10.853271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:10.853271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:10.853271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:10.853271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:10.853271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:10.853271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:10.853271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:10.853271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:10.853271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:10.853271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:10.853271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:10.853271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:10.853271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:10.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:10.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:10.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:10.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:10.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:10.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:10.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:10.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:10.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:10.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:10.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:10.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:10.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:10.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:10.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:10.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:10.854431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:10.855356 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:10.855356 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:10.855356 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:10.855356 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 292/584] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:10.855579 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49275aed-1680-4a9b-bdfe-f32a154f207a None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:10.869219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645f888d-1bae-40d7-a239-c14b200190d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:10.869856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645f888d-1bae-40d7-a239-c14b200190d4 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:10.870134 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645f888d-1bae-40d7-a239-c14b200190d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:10.870404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645f888d-1bae-40d7-a239-c14b200190d4 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:10.871051 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645f888d-1bae-40d7-a239-c14b200190d4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:10.871357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645f888d-1bae-40d7-a239-c14b200190d4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:10.872717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645f888d-1bae-40d7-a239-c14b200190d4 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:10.889325 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 293/585] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:10.889325 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645f888d-1bae-40d7-a239-c14b200190d4 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:10.932137 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:10.969897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49275aed-1680-4a9b-bdfe-f32a154f207a None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:10.995715 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645f888d-1bae-40d7-a239-c14b200190d4 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:11.014678 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c65087-5ecd-47a9-9eba-28973ce73f12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:11.015665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c65087-5ecd-47a9-9eba-28973ce73f12 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:11.015945 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c65087-5ecd-47a9-9eba-28973ce73f12 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:11.016217 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c65087-5ecd-47a9-9eba-28973ce73f12 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:11.026240 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c65087-5ecd-47a9-9eba-28973ce73f12 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:11.028138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c65087-5ecd-47a9-9eba-28973ce73f12 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:11.032863 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa48143-0cbb-400d-b107-b924899e4b52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:11.033458 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa48143-0cbb-400d-b107-b924899e4b52 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:11.033740 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa48143-0cbb-400d-b107-b924899e4b52 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:11.034013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa48143-0cbb-400d-b107-b924899e4b52 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:11.035489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa48143-0cbb-400d-b107-b924899e4b52 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:11.035791 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa48143-0cbb-400d-b107-b924899e4b52 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:11.037170 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa48143-0cbb-400d-b107-b924899e4b52 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:11.040309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-64c65087-5ecd-47a9-9eba-28973ce73f12 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 04 00:47:11.040309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:11.040309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:11.040309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:11.040309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:11.040309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:11.040309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:11.040309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:11.040309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:11.040309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:11.040309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:11.040309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:11.040309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:11.040309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:11.040309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:11.040309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:11.041255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:11.042235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:11.042235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:11.042235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:11.042235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:11.042235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:11.042235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:11.042235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:11.042235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:11.042235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 04 00:47:11.042235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:11.046273 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 293/586] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:10 2022] GET /identity/v3/users/neutron => generated 85 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:11.046936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c65087-5ecd-47a9-9eba-28973ce73f12 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:11.049020 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 294/587] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:11.050416 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa48143-0cbb-400d-b107-b924899e4b52 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:11.077961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:11.157969 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:11.160944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa48143-0cbb-400d-b107-b924899e4b52 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:11.169233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c65087-5ecd-47a9-9eba-28973ce73f12 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:11.199334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dc6f79-1ed3-4a5a-9fad-106ae8f6d3c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:11.200047 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc6f79-1ed3-4a5a-9fad-106ae8f6d3c6 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:11.200392 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc6f79-1ed3-4a5a-9fad-106ae8f6d3c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:11.200732 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc6f79-1ed3-4a5a-9fad-106ae8f6d3c6 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:11.205831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079d64ab-9445-49ca-9645-e7a9f2350b15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:11.207094 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079d64ab-9445-49ca-9645-e7a9f2350b15 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:11.207373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079d64ab-9445-49ca-9645-e7a9f2350b15 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:11.207644 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079d64ab-9445-49ca-9645-e7a9f2350b15 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:11.208288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079d64ab-9445-49ca-9645-e7a9f2350b15 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:11.208591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079d64ab-9445-49ca-9645-e7a9f2350b15 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:11.209189 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dc6f79-1ed3-4a5a-9fad-106ae8f6d3c6 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:11.209984 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079d64ab-9445-49ca-9645-e7a9f2350b15 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:11.210732 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dc6f79-1ed3-4a5a-9fad-106ae8f6d3c6 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:11.222144 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 294/588] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:11.222767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079d64ab-9445-49ca-9645-e7a9f2350b15 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:11.227354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-a2dc6f79-1ed3-4a5a-9fad-106ae8f6d3c6 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 04 00:47:11.227354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:11.227354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:11.227354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:11.227354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:11.227354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:11.227354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:11.227354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:11.227354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:11.227354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:11.227354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:11.227354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:11.227354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:11.227354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:11.227354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:11.227354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:11.233184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:11.234457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:11.234457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:11.234457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:11.234457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:11.234457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:11.234457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:11.234457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:11.234457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:11.234457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 04 00:47:11.234457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:11.234457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 295/589] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:47:11 2022] GET /identity/v3/users/cinder => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:11.234457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dc6f79-1ed3-4a5a-9fad-106ae8f6d3c6 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:11.320574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:11.337540 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dc6f79-1ed3-4a5a-9fad-106ae8f6d3c6 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:11.340448 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079d64ab-9445-49ca-9645-e7a9f2350b15 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:11.372209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8d00cc-f134-412d-a5a8-df83c45fb97e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:11.373174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8d00cc-f134-412d-a5a8-df83c45fb97e None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:11.373450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8d00cc-f134-412d-a5a8-df83c45fb97e None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:11.373728 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8d00cc-f134-412d-a5a8-df83c45fb97e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:11.380319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef29758-29ac-41d0-8e72-ca0889ca37d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:11.381131 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8d00cc-f134-412d-a5a8-df83c45fb97e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:11.381253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef29758-29ac-41d0-8e72-ca0889ca37d6 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:11.381661 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef29758-29ac-41d0-8e72-ca0889ca37d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:11.382084 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef29758-29ac-41d0-8e72-ca0889ca37d6 None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:11.382581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8d00cc-f134-412d-a5a8-df83c45fb97e None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:11.383150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef29758-29ac-41d0-8e72-ca0889ca37d6 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:11.383607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef29758-29ac-41d0-8e72-ca0889ca37d6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:11.385490 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef29758-29ac-41d0-8e72-ca0889ca37d6 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:11.394277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-be8d00cc-f134-412d-a5a8-df83c45fb97e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:11.394277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:11.394277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:11.394277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:11.394277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:11.394277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:11.394277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:11.394277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:11.394277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:11.394277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:11.394277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:11.394277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:11.394277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:11.394277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:11.394277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:11.395451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:11.395451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:11.395451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:11.395451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:11.395451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:11.395451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:11.395451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:11.395451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:11.395451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:11.395451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:11.395451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:11.395451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:11.395451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:11.395451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:11.395451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:11.395451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:11.395451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:11.396352 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:11.396352 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:11.396352 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:11.396352 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:11.396352 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:11.396352 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:11.396352 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:11.396352 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:11.396352 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 296/590] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:11.396856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8d00cc-f134-412d-a5a8-df83c45fb97e None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:11.414242 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 295/591] 10.222.0.42 () {56 vars in 1139 bytes} [Sun Sep 4 00:47:11 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:11.414242 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef29758-29ac-41d0-8e72-ca0889ca37d6 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:11.423379 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:11.509687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8d00cc-f134-412d-a5a8-df83c45fb97e None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:11.533255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef29758-29ac-41d0-8e72-ca0889ca37d6 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:11.548390 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a124ca1b-f8a9-42a6-8849-729c4b4ad057 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:11.549013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a124ca1b-f8a9-42a6-8849-729c4b4ad057 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:11.549326 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a124ca1b-f8a9-42a6-8849-729c4b4ad057 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:11.549595 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a124ca1b-f8a9-42a6-8849-729c4b4ad057 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:11.556896 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a124ca1b-f8a9-42a6-8849-729c4b4ad057 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:11.558364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a124ca1b-f8a9-42a6-8849-729c4b4ad057 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:11.570809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-a124ca1b-f8a9-42a6-8849-729c4b4ad057 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:11.570809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:11.570809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:11.570809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:11.570809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:11.570809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:11.570809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:11.570809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:11.570809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:11.570809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:11.570809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:11.570809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:11.570809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:11.570809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:11.570809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:11.571678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:11.571678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:11.571678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:11.571678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:11.571678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:11.571678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:11.571678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:11.571678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:11.571678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:11.571678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:11.571678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:11.571678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:11.571678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:11.571678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:11.571678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:11.571678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:11.571678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:11.572603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:11.572603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:11.572603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:11.572603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:11.572603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:11.572603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:11.572603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:11.572603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:11.572603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 297/592] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:11.572603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a124ca1b-f8a9-42a6-8849-729c4b4ad057 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:11.577188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d78602-2a5d-4605-8aeb-6a481fa47b88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:11.577817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d78602-2a5d-4605-8aeb-6a481fa47b88 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:11.578100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d78602-2a5d-4605-8aeb-6a481fa47b88 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:11.578377 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d78602-2a5d-4605-8aeb-6a481fa47b88 None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:11.579117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d78602-2a5d-4605-8aeb-6a481fa47b88 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:11.579427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d78602-2a5d-4605-8aeb-6a481fa47b88 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:11.580845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d78602-2a5d-4605-8aeb-6a481fa47b88 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:11.591727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:11.607939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 296/593] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Sep 4 00:47:11 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:11.607939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d78602-2a5d-4605-8aeb-6a481fa47b88 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:11.664522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:11.689451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a124ca1b-f8a9-42a6-8849-729c4b4ad057 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:11.714213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d78602-2a5d-4605-8aeb-6a481fa47b88 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:11.728543 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4d047f-2973-4bea-8391-dcefb1f01e38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:11.729199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4d047f-2973-4bea-8391-dcefb1f01e38 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:11.729477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4d047f-2973-4bea-8391-dcefb1f01e38 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:11.729752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4d047f-2973-4bea-8391-dcefb1f01e38 None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:11.730435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4d047f-2973-4bea-8391-dcefb1f01e38 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:11.730741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4d047f-2973-4bea-8391-dcefb1f01e38 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:11.732155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4d047f-2973-4bea-8391-dcefb1f01e38 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:11.744703 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 298/594] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:11.745547 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4d047f-2973-4bea-8391-dcefb1f01e38 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:11.756155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a08783-af42-4482-b248-0e633f60f813 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:11.756782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a08783-af42-4482-b248-0e633f60f813 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:11.757059 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a08783-af42-4482-b248-0e633f60f813 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:11.757357 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a08783-af42-4482-b248-0e633f60f813 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:11.767688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a08783-af42-4482-b248-0e633f60f813 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:11.769333 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a08783-af42-4482-b248-0e633f60f813 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:11.784774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-60a08783-af42-4482-b248-0e633f60f813 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:11.784774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:11.784774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:11.784774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:11.784774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:11.784774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:11.784774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:11.784774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:11.784774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:11.784774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:11.784774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:11.784774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:11.784774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:11.784774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:11.784774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:11.784774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:11.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:11.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:11.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:11.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:11.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:11.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:11.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:11.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:11.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:11.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:11.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:11.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:11.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:11.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:11.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:11.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:11.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:11.786656 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:11.786656 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:11.786656 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:11.790682 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 297/595] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:11.791518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a08783-af42-4482-b248-0e633f60f813 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:11.843780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:11.853508 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4d047f-2973-4bea-8391-dcefb1f01e38 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:11.890054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95215704-4ea1-4989-a6d3-bb90672f3cec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:11.890670 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95215704-4ea1-4989-a6d3-bb90672f3cec None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:11.890952 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95215704-4ea1-4989-a6d3-bb90672f3cec None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:11.891227 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95215704-4ea1-4989-a6d3-bb90672f3cec None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:11.891919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95215704-4ea1-4989-a6d3-bb90672f3cec None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:11.892231 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95215704-4ea1-4989-a6d3-bb90672f3cec None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:11.893764 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95215704-4ea1-4989-a6d3-bb90672f3cec None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:11.898162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a08783-af42-4482-b248-0e633f60f813 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:11.903869 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:11.913106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 299/596] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:11.913106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95215704-4ea1-4989-a6d3-bb90672f3cec None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:11.933467 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68a4bc7-22c4-46f2-afd7-5a7a75150b61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:11.934096 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68a4bc7-22c4-46f2-afd7-5a7a75150b61 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:11.934380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68a4bc7-22c4-46f2-afd7-5a7a75150b61 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:11.934656 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68a4bc7-22c4-46f2-afd7-5a7a75150b61 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:11.945508 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68a4bc7-22c4-46f2-afd7-5a7a75150b61 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:11.946946 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68a4bc7-22c4-46f2-afd7-5a7a75150b61 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:11.961955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-d68a4bc7-22c4-46f2-afd7-5a7a75150b61 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:11.961955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:11.961955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:11.961955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:11.961955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:11.961955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:11.961955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:11.961955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:11.961955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:11.961955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:11.961955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:11.961955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:11.961955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:11.961955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:11.961955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:11.961955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:11.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:11.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:11.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:11.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:11.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:11.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:11.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:11.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:11.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:11.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:11.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:11.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:11.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:11.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:11.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:11.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:11.962908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:11.963900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:11.963900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:11.963900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:11.963900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 298/597] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:11.963900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68a4bc7-22c4-46f2-afd7-5a7a75150b61 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:12.027828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95215704-4ea1-4989-a6d3-bb90672f3cec None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:12.068032 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68a4bc7-22c4-46f2-afd7-5a7a75150b61 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:12.069173 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a40742c-faf1-4dbc-85aa-fef12f6c3261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:12.069602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:12.070560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a40742c-faf1-4dbc-85aa-fef12f6c3261 None admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:12.070839 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a40742c-faf1-4dbc-85aa-fef12f6c3261 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:12.071109 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a40742c-faf1-4dbc-85aa-fef12f6c3261 None admin] PATH_INFO: `/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/69153344fc4b45588bdbcecdfcfd7c3f/roles/df912a77fbaa479db172f76a214f8bc8` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:12.087184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a40742c-faf1-4dbc-85aa-fef12f6c3261 None admin] RBAC: Authorizing `identity:create_grant(project_id=b1f56314d44d4faaa88e735ea196783f, user_id=69153344fc4b45588bdbcecdfcfd7c3f, role_id=df912a77fbaa479db172f76a214f8bc8)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:12.089140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a40742c-faf1-4dbc-85aa-fef12f6c3261 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:12.104881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7546f596-df06-48c3-a870-d59d5f258ff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:12.104881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546f596-df06-48c3-a870-d59d5f258ff0 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:12.104881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546f596-df06-48c3-a870-d59d5f258ff0 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:12.105333 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546f596-df06-48c3-a870-d59d5f258ff0 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:12.105792 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7546f596-df06-48c3-a870-d59d5f258ff0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:12.106093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7546f596-df06-48c3-a870-d59d5f258ff0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:12.107432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7546f596-df06-48c3-a870-d59d5f258ff0 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:12.119039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 299/598] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:12.119648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7546f596-df06-48c3-a870-d59d5f258ff0 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:12.153346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 300/599] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:47:11 2022] PUT /identity/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/69153344fc4b45588bdbcecdfcfd7c3f/roles/df912a77fbaa479db172f76a214f8bc8 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:47:12.154159 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a40742c-faf1-4dbc-85aa-fef12f6c3261 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:12.212954 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:12.270042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a40742c-faf1-4dbc-85aa-fef12f6c3261 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:12.273351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7546f596-df06-48c3-a870-d59d5f258ff0 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:12.310168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f423ccd-1bdf-422e-97e4-3505f016f074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:12.310788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f423ccd-1bdf-422e-97e4-3505f016f074 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:12.311065 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f423ccd-1bdf-422e-97e4-3505f016f074 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:12.311332 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f423ccd-1bdf-422e-97e4-3505f016f074 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:12.311981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f423ccd-1bdf-422e-97e4-3505f016f074 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:12.312282 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f423ccd-1bdf-422e-97e4-3505f016f074 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:12.313692 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f423ccd-1bdf-422e-97e4-3505f016f074 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:12.323272 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb081b4a-dd56-4103-a88a-de45ad7a7478 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:12.324290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb081b4a-dd56-4103-a88a-de45ad7a7478 None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:12.324816 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb081b4a-dd56-4103-a88a-de45ad7a7478 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:12.325327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb081b4a-dd56-4103-a88a-de45ad7a7478 None admin] PATH_INFO: `/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/57c88f16c7df40adb917f49804fda21f/roles/df912a77fbaa479db172f76a214f8bc8` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:12.327288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 301/600] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:12.327959 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f423ccd-1bdf-422e-97e4-3505f016f074 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:12.348306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb081b4a-dd56-4103-a88a-de45ad7a7478 None admin] RBAC: Authorizing `identity:create_grant(project_id=b1f56314d44d4faaa88e735ea196783f, user_id=57c88f16c7df40adb917f49804fda21f, role_id=df912a77fbaa479db172f76a214f8bc8)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:12.349797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb081b4a-dd56-4103-a88a-de45ad7a7478 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:12.445332 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 300/601] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:47:12 2022] PUT /identity/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/57c88f16c7df40adb917f49804fda21f/roles/df912a77fbaa479db172f76a214f8bc8 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:47:12.446096 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb081b4a-dd56-4103-a88a-de45ad7a7478 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:12.455064 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f423ccd-1bdf-422e-97e4-3505f016f074 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:12.469749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:12.472993 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:12.507976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d787dff9-059e-4301-bd69-3079efec8fa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:12.508782 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d787dff9-059e-4301-bd69-3079efec8fa5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:12.509204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d787dff9-059e-4301-bd69-3079efec8fa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:12.509642 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d787dff9-059e-4301-bd69-3079efec8fa5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:12.517663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d787dff9-059e-4301-bd69-3079efec8fa5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:12.519608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d787dff9-059e-4301-bd69-3079efec8fa5 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:12.532380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-d787dff9-059e-4301-bd69-3079efec8fa5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:12.532380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:12.532380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:12.532380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:12.532380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:12.532380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:12.532380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:12.532380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:12.532380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:12.532380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:12.532380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:12.532380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:12.532380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:12.532380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:12.532380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:12.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:12.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:12.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:12.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:12.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:12.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:12.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:12.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:12.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:12.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:12.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:12.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:12.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:12.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:12.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:12.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:12.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:12.535313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:12.535313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:12.535313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:12.535313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:12.535313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:12.535313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:12.535313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:12.535313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:12.541149 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 302/602] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:12.543108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011ba768-0d94-4edc-976d-e5312f10a7d1 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:12.543398 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011ba768-0d94-4edc-976d-e5312f10a7d1 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:12.543676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011ba768-0d94-4edc-976d-e5312f10a7d1 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:12.545440 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 303/603] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:12 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:12.546212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011ba768-0d94-4edc-976d-e5312f10a7d1 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:12.583067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb081b4a-dd56-4103-a88a-de45ad7a7478 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:12.630708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e6897f-069c-4ec5-bc0f-6855fee4ee31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:12.632215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e6897f-069c-4ec5-bc0f-6855fee4ee31 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:12.632500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e6897f-069c-4ec5-bc0f-6855fee4ee31 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:12.632836 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e6897f-069c-4ec5-bc0f-6855fee4ee31 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:12.641460 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e6897f-069c-4ec5-bc0f-6855fee4ee31 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:12.643481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e6897f-069c-4ec5-bc0f-6855fee4ee31 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:12.656278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-f5e6897f-069c-4ec5-bc0f-6855fee4ee31 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:12.656278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:12.656278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:12.656278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:12.656278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:12.656278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:12.656278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:12.656278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:12.656278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:12.656278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:12.656278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:12.656278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:12.656278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:12.656278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:12.656278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:12.657266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:12.657266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:12.657266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:12.657266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:12.657266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:12.657266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:12.657266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:12.657266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:12.657266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:12.657266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:12.657266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:12.657266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:12.657266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:12.657266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:12.657266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:12.657266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:12.657266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:12.658155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:12.658155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:12.658155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:12.658155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:12.658155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:12.658155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:12.658155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:12.658155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:12.661241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 301/604] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:12.662714 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6d4cf5-a186-4e7b-8b68-3d6bc8ed6933 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:12.663237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6d4cf5-a186-4e7b-8b68-3d6bc8ed6933 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:12.663669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6d4cf5-a186-4e7b-8b68-3d6bc8ed6933 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:12.668297 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011ba768-0d94-4edc-976d-e5312f10a7d1 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:12.683965 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-aa6d4cf5-a186-4e7b-8b68-3d6bc8ed6933 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:12.685538 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-aa6d4cf5-a186-4e7b-8b68-3d6bc8ed6933 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:12.704586 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d411bd9d-6e1e-42ac-96ac-8ffbb82d435f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:12.705319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d411bd9d-6e1e-42ac-96ac-8ffbb82d435f None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:12.705883 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d411bd9d-6e1e-42ac-96ac-8ffbb82d435f None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:12.706288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d411bd9d-6e1e-42ac-96ac-8ffbb82d435f None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:12.707291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d411bd9d-6e1e-42ac-96ac-8ffbb82d435f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:12.707692 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d411bd9d-6e1e-42ac-96ac-8ffbb82d435f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:12.709768 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d411bd9d-6e1e-42ac-96ac-8ffbb82d435f None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:12.723858 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 304/605] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:12.724966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d411bd9d-6e1e-42ac-96ac-8ffbb82d435f None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:12.758042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:12.773206 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 302/606] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:12.774291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6d4cf5-a186-4e7b-8b68-3d6bc8ed6933 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:12.780666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:12.891440 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d411bd9d-6e1e-42ac-96ac-8ffbb82d435f None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:12.963391 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe68695-11be-4403-8a8c-ef6231aa0713 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:12.964826 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe68695-11be-4403-8a8c-ef6231aa0713 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:12.965409 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe68695-11be-4403-8a8c-ef6231aa0713 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:12.965904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe68695-11be-4403-8a8c-ef6231aa0713 None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:12.967842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe68695-11be-4403-8a8c-ef6231aa0713 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:12.968970 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe68695-11be-4403-8a8c-ef6231aa0713 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:12.970593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe68695-11be-4403-8a8c-ef6231aa0713 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:12.975633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6d4cf5-a186-4e7b-8b68-3d6bc8ed6933 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:12.993910 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 305/607] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:13.000980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe68695-11be-4403-8a8c-ef6231aa0713 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:13.054718 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e75def-5584-4e19-8c35-eb017afa5ddb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:13.055394 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e75def-5584-4e19-8c35-eb017afa5ddb None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:13.055673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e75def-5584-4e19-8c35-eb017afa5ddb None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:13.055938 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e75def-5584-4e19-8c35-eb017afa5ddb None admin] PATH_INFO: `/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/da4f44b86ca94adf959d9a41d915c902/roles/df912a77fbaa479db172f76a214f8bc8` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:13.089912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e75def-5584-4e19-8c35-eb017afa5ddb None admin] RBAC: Authorizing `identity:create_grant(project_id=b1f56314d44d4faaa88e735ea196783f, user_id=da4f44b86ca94adf959d9a41d915c902, role_id=df912a77fbaa479db172f76a214f8bc8)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:13.091520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e75def-5584-4e19-8c35-eb017afa5ddb None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:13.110606 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:13.120552 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe68695-11be-4403-8a8c-ef6231aa0713 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:13.153812 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc731577-b948-4684-9c22-d419193716b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:13.154461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc731577-b948-4684-9c22-d419193716b3 None admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:13.154739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc731577-b948-4684-9c22-d419193716b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:13.155009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc731577-b948-4684-9c22-d419193716b3 None admin] PATH_INFO: `/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/615a607b6be14b82979f94eabaf4c34c/roles/df912a77fbaa479db172f76a214f8bc8` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:13.170452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc731577-b948-4684-9c22-d419193716b3 None admin] RBAC: Authorizing `identity:create_grant(project_id=b1f56314d44d4faaa88e735ea196783f, user_id=615a607b6be14b82979f94eabaf4c34c, role_id=df912a77fbaa479db172f76a214f8bc8)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:13.171908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc731577-b948-4684-9c22-d419193716b3 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:13.189107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 303/608] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:47:12 2022] PUT /identity/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/da4f44b86ca94adf959d9a41d915c902/roles/df912a77fbaa479db172f76a214f8bc8 => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:47:13.189884 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e75def-5584-4e19-8c35-eb017afa5ddb None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:13.259788 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:13.267686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 306/609] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:47:13 2022] PUT /identity/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/615a607b6be14b82979f94eabaf4c34c/roles/df912a77fbaa479db172f76a214f8bc8 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 00:47:13.316506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e75def-5584-4e19-8c35-eb017afa5ddb None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:13.355979 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea92dce7-48b5-44d2-ad97-0b6f57896288 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:13.357098 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea92dce7-48b5-44d2-ad97-0b6f57896288 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:13.357954 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea92dce7-48b5-44d2-ad97-0b6f57896288 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:13.358454 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea92dce7-48b5-44d2-ad97-0b6f57896288 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:13.371063 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea92dce7-48b5-44d2-ad97-0b6f57896288 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:13.378362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-ea92dce7-48b5-44d2-ad97-0b6f57896288 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:13.378362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:13.378362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:13.378362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:13.378362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:13.378362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:13.378362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:13.378362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:13.378362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:13.378362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:13.378362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:13.378362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:13.378362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:13.378362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:13.378362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:13.378362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:13.379339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:13.379339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:13.379339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:13.379339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:13.379339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:13.379339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:13.379339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:13.379339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:13.379339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:13.379339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:13.379339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:13.379339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:13.379339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:13.379339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:13.379339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:13.379339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:13.380146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:13.380146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:13.380146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:13.380146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:13.380146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:13.380146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:13.380146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:13.380146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:13.380146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:13.380146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:13.380146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:13.380146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:13.387742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 304/610] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:13.389487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc731577-b948-4684-9c22-d419193716b3 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:13.506233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc731577-b948-4684-9c22-d419193716b3 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:13.550216 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77da101-79d2-4c8b-99bb-a6471bee5944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:13.550752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77da101-79d2-4c8b-99bb-a6471bee5944 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:13.551029 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77da101-79d2-4c8b-99bb-a6471bee5944 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:13.551308 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77da101-79d2-4c8b-99bb-a6471bee5944 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:13.551982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77da101-79d2-4c8b-99bb-a6471bee5944 None admin] RBAC: Adding query filter params (name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:13.552283 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77da101-79d2-4c8b-99bb-a6471bee5944 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:13.553687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77da101-79d2-4c8b-99bb-a6471bee5944 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:13.563510 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 307/611] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:13.569582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea92dce7-48b5-44d2-ad97-0b6f57896288 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:13.679863 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea92dce7-48b5-44d2-ad97-0b6f57896288 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:13.694931 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:13.717631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b212509d-9578-44cb-8782-e6ed06ae32eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:13.718447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b212509d-9578-44cb-8782-e6ed06ae32eb None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:13.718828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b212509d-9578-44cb-8782-e6ed06ae32eb None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:13.719196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b212509d-9578-44cb-8782-e6ed06ae32eb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:13.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b212509d-9578-44cb-8782-e6ed06ae32eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:13.732563 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b212509d-9578-44cb-8782-e6ed06ae32eb None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:13.748269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-b212509d-9578-44cb-8782-e6ed06ae32eb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:13.748269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:13.748269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:13.748269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:13.748269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:13.748269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:13.748269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:13.748269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:13.748269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:13.748269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:13.748269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:13.748269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:13.748269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:13.748269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:13.748269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:13.748269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:13.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:13.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:13.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:13.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:13.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:13.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:13.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:13.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:13.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:13.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:13.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:13.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:13.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:13.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:13.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:13.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:13.749407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:13.750483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:13.750483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:13.750483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:13.756704 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 305/612] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:13.758150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77da101-79d2-4c8b-99bb-a6471bee5944 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:13.767211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:13.918761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77da101-79d2-4c8b-99bb-a6471bee5944 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:13.965784 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794e6a01-3f7e-4298-8ca4-a764af2c3579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:13.966398 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794e6a01-3f7e-4298-8ca4-a764af2c3579 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:13.966680 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794e6a01-3f7e-4298-8ca4-a764af2c3579 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:13.966956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794e6a01-3f7e-4298-8ca4-a764af2c3579 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:13.967611 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794e6a01-3f7e-4298-8ca4-a764af2c3579 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:13.967916 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794e6a01-3f7e-4298-8ca4-a764af2c3579 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:13.969327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794e6a01-3f7e-4298-8ca4-a764af2c3579 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:13.984015 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 308/613] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:13.990632 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b212509d-9578-44cb-8782-e6ed06ae32eb None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:14.160342 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b212509d-9578-44cb-8782-e6ed06ae32eb None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:14.205886 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8681b54-6d4c-450e-b32a-78f971309be3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:14.206505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8681b54-6d4c-450e-b32a-78f971309be3 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:14.206786 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8681b54-6d4c-450e-b32a-78f971309be3 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:14.207059 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8681b54-6d4c-450e-b32a-78f971309be3 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:14.219113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8681b54-6d4c-450e-b32a-78f971309be3 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:14.220534 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8681b54-6d4c-450e-b32a-78f971309be3 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:14.236115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-e8681b54-6d4c-450e-b32a-78f971309be3 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 04 00:47:14.236115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:14.236115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:14.236115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:14.236115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:14.236115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:14.236115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:14.236115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:14.236115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:14.236115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:14.236115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:14.236115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:14.236115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:14.236115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:14.236115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:14.236115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:14.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:14.238963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:14.238963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:14.238963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:14.238963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:14.238963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:14.238963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:14.238963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:14.238963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:14.238963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 04 00:47:14.238963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:14.238963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 306/614] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:47:13 2022] GET /identity/v3/users/swift => generated 83 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:14.245487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794e6a01-3f7e-4298-8ca4-a764af2c3579 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:14.276529 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25685b6-b1c2-4c27-81d5-e14f8c6cca92 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:14.276843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25685b6-b1c2-4c27-81d5-e14f8c6cca92 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:14.277111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25685b6-b1c2-4c27-81d5-e14f8c6cca92 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:14.278682 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 307/615] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:47:14.283591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7a3699-82dc-4583-85d5-6cf5e1f67678 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:14.283941 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7a3699-82dc-4583-85d5-6cf5e1f67678 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:14.284197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7a3699-82dc-4583-85d5-6cf5e1f67678 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:14.303155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-4f7a3699-82dc-4583-85d5-6cf5e1f67678 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:14.304350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-4f7a3699-82dc-4583-85d5-6cf5e1f67678 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:14.323324 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:14.346399 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:14.379524 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794e6a01-3f7e-4298-8ca4-a764af2c3579 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:14.425106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f9da0b-7b3e-40ef-be3f-c8cb33dbbd1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:14.425106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f9da0b-7b3e-40ef-be3f-c8cb33dbbd1f None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:14.425106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f9da0b-7b3e-40ef-be3f-c8cb33dbbd1f None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:14.425106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f9da0b-7b3e-40ef-be3f-c8cb33dbbd1f None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:14.425740 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f9da0b-7b3e-40ef-be3f-c8cb33dbbd1f None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:14.425740 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f9da0b-7b3e-40ef-be3f-c8cb33dbbd1f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:14.425740 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f9da0b-7b3e-40ef-be3f-c8cb33dbbd1f None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:14.435654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 308/616] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:47:14.452361 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 309/617] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Sep 4 00:47:14 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:14.458244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7a3699-82dc-4583-85d5-6cf5e1f67678 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:14.604835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7a3699-82dc-4583-85d5-6cf5e1f67678 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:14.656093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4450e9c4-9cfb-4e54-950e-94edfaf69157 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:14.656735 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4450e9c4-9cfb-4e54-950e-94edfaf69157 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:14.657015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4450e9c4-9cfb-4e54-950e-94edfaf69157 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:14.657319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4450e9c4-9cfb-4e54-950e-94edfaf69157 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:14.671502 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4450e9c4-9cfb-4e54-950e-94edfaf69157 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:14.672958 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4450e9c4-9cfb-4e54-950e-94edfaf69157 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:14.692821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-4450e9c4-9cfb-4e54-950e-94edfaf69157 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:14.692821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:14.692821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:14.692821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:14.692821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:14.692821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:14.692821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:14.692821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:14.692821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:14.692821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:14.692821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:14.692821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:14.692821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:14.692821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:14.692821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:14.692821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:14.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:14.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:14.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:14.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:14.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:14.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:14.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:14.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:14.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:14.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:14.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:14.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:14.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:14.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:14.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:14.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:14.694577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:14.695616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:14.695616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:14.695616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:14.695616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 309/618] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:14.701809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f9da0b-7b3e-40ef-be3f-c8cb33dbbd1f None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:14.809049 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f9da0b-7b3e-40ef-be3f-c8cb33dbbd1f None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:14.830525 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:14.834984 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4450e9c4-9cfb-4e54-950e-94edfaf69157 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:14.849097 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:14.849272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff303fde-3e2a-4a00-8096-1edc21e496dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:14.849909 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff303fde-3e2a-4a00-8096-1edc21e496dd None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:14.850192 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff303fde-3e2a-4a00-8096-1edc21e496dd None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:14.850666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff303fde-3e2a-4a00-8096-1edc21e496dd None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:14.851738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff303fde-3e2a-4a00-8096-1edc21e496dd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:14.852048 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff303fde-3e2a-4a00-8096-1edc21e496dd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:14.853446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff303fde-3e2a-4a00-8096-1edc21e496dd None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:14.866783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 310/619] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:14.868104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6155415f-ce62-4392-ab71-f4f639bd3df4 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:14.868427 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6155415f-ce62-4392-ab71-f4f639bd3df4 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:14.868677 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6155415f-ce62-4392-ab71-f4f639bd3df4 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:14.874592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 311/620] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:14 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:14.875554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6155415f-ce62-4392-ab71-f4f639bd3df4 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:14.961812 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4450e9c4-9cfb-4e54-950e-94edfaf69157 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:14.980118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6155415f-ce62-4392-ab71-f4f639bd3df4 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:14.999234 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d415347-a450-4f60-a3d0-233f1ecd38ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:14.999871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d415347-a450-4f60-a3d0-233f1ecd38ee None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:15.000152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d415347-a450-4f60-a3d0-233f1ecd38ee None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:15.000426 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d415347-a450-4f60-a3d0-233f1ecd38ee None admin] PATH_INFO: `/v3/roles/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:15.011736 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d415347-a450-4f60-a3d0-233f1ecd38ee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:15.015236 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec96cbc-4f13-4644-9b2b-23be20c3e07c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:15.015867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec96cbc-4f13-4644-9b2b-23be20c3e07c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:15.016142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec96cbc-4f13-4644-9b2b-23be20c3e07c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:15.016408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec96cbc-4f13-4644-9b2b-23be20c3e07c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:15.017495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-0d415347-a450-4f60-a3d0-233f1ecd38ee None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:15.017495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:15.017495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:15.017495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:15.017495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:15.017495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:15.017495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:15.017495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:15.017495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:15.017495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:15.017495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:15.017495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:15.017495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:15.017495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:15.017495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:15.017495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:15.019041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:15.019041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:15.019041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:15.019041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:15.019041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:15.019041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:15.019041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:15.019041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:15.019041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:15.019041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:15.019041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:15.019041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:15.019041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:15.019041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:15.019041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:15.019041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:15.020832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:15.020832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:15.020832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:15.020832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:15.020832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:15.020832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:15.020832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:15.020832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:15.020832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:15.020832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:15.020832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:15.020832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:15.020832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 310/621] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:15.022479 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fd6282-77fb-495e-8c6a-4882b37fe830 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:15.022479 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fd6282-77fb-495e-8c6a-4882b37fe830 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:15.022479 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fd6282-77fb-495e-8c6a-4882b37fe830 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:15.028938 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec96cbc-4f13-4644-9b2b-23be20c3e07c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:15.030499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec96cbc-4f13-4644-9b2b-23be20c3e07c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:15.040763 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-e6fd6282-77fb-495e-8c6a-4882b37fe830 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:15.041994 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e6fd6282-77fb-495e-8c6a-4882b37fe830 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:15.043056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-5ec96cbc-4f13-4644-9b2b-23be20c3e07c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:15.043056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:15.043056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:15.043056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:15.043056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:15.043056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:15.043056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:15.043056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:15.043056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:15.043056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:15.043056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:15.043056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:15.043056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:15.043056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:15.043056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:15.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:15.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:15.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:15.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:15.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:15.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:15.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:15.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:15.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:15.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:15.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:15.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:15.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:15.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:15.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:15.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:15.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:15.046389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:15.046389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:15.046389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:15.046389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:15.046389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:15.046389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:15.046389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:15.046389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:15.046389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 312/622] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:15.046389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec96cbc-4f13-4644-9b2b-23be20c3e07c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:15.089921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:15.108579 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:15.124502 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 311/623] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:15.125882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fd6282-77fb-495e-8c6a-4882b37fe830 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:15.157212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec96cbc-4f13-4644-9b2b-23be20c3e07c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:15.188044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0973e0d1-bfef-4394-8912-dbd59fa66607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:15.188811 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0973e0d1-bfef-4394-8912-dbd59fa66607 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:15.189217 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0973e0d1-bfef-4394-8912-dbd59fa66607 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:15.189589 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0973e0d1-bfef-4394-8912-dbd59fa66607 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:15.190404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0973e0d1-bfef-4394-8912-dbd59fa66607 None admin] RBAC: Adding query filter params (name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:15.190803 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0973e0d1-bfef-4394-8912-dbd59fa66607 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:15.192260 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0973e0d1-bfef-4394-8912-dbd59fa66607 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:15.202399 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 313/624] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:15 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:15.214276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0973e0d1-bfef-4394-8912-dbd59fa66607 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:15.307361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fd6282-77fb-495e-8c6a-4882b37fe830 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:15.321622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0973e0d1-bfef-4394-8912-dbd59fa66607 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:15.323116 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:15.366900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f25faa-2580-4e17-9011-9b6175a79296 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:15.367575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f25faa-2580-4e17-9011-9b6175a79296 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:15.367853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f25faa-2580-4e17-9011-9b6175a79296 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:15.368179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f25faa-2580-4e17-9011-9b6175a79296 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:15.380712 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f25faa-2580-4e17-9011-9b6175a79296 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:15.382207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f25faa-2580-4e17-9011-9b6175a79296 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:15.383662 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235783eb-f7b9-4c54-897b-56e9f59ca197 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:15.384577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235783eb-f7b9-4c54-897b-56e9f59ca197 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:15.385179 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235783eb-f7b9-4c54-897b-56e9f59ca197 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:15.385437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235783eb-f7b9-4c54-897b-56e9f59ca197 None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:15.386540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235783eb-f7b9-4c54-897b-56e9f59ca197 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:15.387079 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235783eb-f7b9-4c54-897b-56e9f59ca197 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:15.396918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235783eb-f7b9-4c54-897b-56e9f59ca197 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:15.398190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-89f25faa-2580-4e17-9011-9b6175a79296 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:15.398190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:15.398190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:15.398190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:15.398190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:15.398190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:15.398190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:15.398190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:15.398190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:15.398190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:15.398190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:15.398190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:15.398190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:15.398190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:15.398190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:15.398190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:15.401893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:15.401893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:15.401893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:15.401893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:15.401893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:15.401893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:15.401893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:15.401893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:15.401893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:15.401893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:15.401893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:15.401893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:15.401893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:15.401893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:15.401893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:15.401893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:15.401893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:15.402756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:15.402756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:15.402756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:15.402756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 314/625] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:15.405647 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f25faa-2580-4e17-9011-9b6175a79296 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:15.430300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 312/626] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:15.438418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235783eb-f7b9-4c54-897b-56e9f59ca197 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:15.515792 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f25faa-2580-4e17-9011-9b6175a79296 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:15.556567 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ba79b6-98dd-4d2d-a5fb-1043808d0366 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:15.557216 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ba79b6-98dd-4d2d-a5fb-1043808d0366 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:15.558026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ba79b6-98dd-4d2d-a5fb-1043808d0366 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:15.558311 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ba79b6-98dd-4d2d-a5fb-1043808d0366 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:15.559250 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ba79b6-98dd-4d2d-a5fb-1043808d0366 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:15.559562 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ba79b6-98dd-4d2d-a5fb-1043808d0366 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:15.561439 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:15.563068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ba79b6-98dd-4d2d-a5fb-1043808d0366 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:15.579589 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 315/627] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:15.586012 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ba79b6-98dd-4d2d-a5fb-1043808d0366 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:15.612208 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235783eb-f7b9-4c54-897b-56e9f59ca197 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:15.642371 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:15.654046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b6e199-2dc8-49a9-b162-ff668fb1e628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:15.654710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b6e199-2dc8-49a9-b162-ff668fb1e628 None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:15.654985 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b6e199-2dc8-49a9-b162-ff668fb1e628 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:15.655255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b6e199-2dc8-49a9-b162-ff668fb1e628 None admin] PATH_INFO: `/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/123ab6ff796c4e599ce1d6f4095bd898/roles/df912a77fbaa479db172f76a214f8bc8` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:15.671097 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b6e199-2dc8-49a9-b162-ff668fb1e628 None admin] RBAC: Authorizing `identity:create_grant(project_id=b1f56314d44d4faaa88e735ea196783f, user_id=123ab6ff796c4e599ce1d6f4095bd898, role_id=df912a77fbaa479db172f76a214f8bc8)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:15.672551 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b6e199-2dc8-49a9-b162-ff668fb1e628 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:15.701607 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ba79b6-98dd-4d2d-a5fb-1043808d0366 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:15.735286 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 313/628] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:47:15 2022] PUT /identity/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/123ab6ff796c4e599ce1d6f4095bd898/roles/df912a77fbaa479db172f76a214f8bc8 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:47:15.736740 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b6e199-2dc8-49a9-b162-ff668fb1e628 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:15.747388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b0052e-fafb-49d8-8225-53e157d493be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:15.749180 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b0052e-fafb-49d8-8225-53e157d493be None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:15.749720 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b0052e-fafb-49d8-8225-53e157d493be None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:15.750168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b0052e-fafb-49d8-8225-53e157d493be None admin] PATH_INFO: `/v3/roles/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:15.766710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b0052e-fafb-49d8-8225-53e157d493be None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:15.774115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-75b0052e-fafb-49d8-8225-53e157d493be None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:15.774115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:15.774115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:15.774115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:15.774115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:15.774115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:15.774115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:15.774115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:15.774115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:15.774115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:15.774115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:15.774115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:15.774115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:15.774115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:15.774115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:15.774115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:15.775388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:15.775388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:15.775388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:15.775388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:15.775388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:15.775388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:15.775388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:15.775388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:15.775388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:15.775388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:15.775388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:15.775388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:15.775388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:15.775388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:15.775388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:15.775388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:15.776445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:15.776445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:15.776445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:15.776445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:15.776445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:15.776445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:15.776445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:15.776445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:15.776445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:15.776445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:15.776445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:15.776445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:15.779063 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 316/629] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:15 2022] GET /identity/v3/roles/service => generated 85 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:15.780876 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d451ee-83e4-4579-aa6d-d332ccb3e645 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:15.781410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d451ee-83e4-4579-aa6d-d332ccb3e645 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:15.781878 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d451ee-83e4-4579-aa6d-d332ccb3e645 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:15.788686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 317/630] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:15 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:15.790032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d451ee-83e4-4579-aa6d-d332ccb3e645 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:15.864687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b6e199-2dc8-49a9-b162-ff668fb1e628 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:15.873716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:15.901175 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:15.914807 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1422eacd-1572-41fc-94e4-e9269029e4a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:15.915595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1422eacd-1572-41fc-94e4-e9269029e4a0 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:15.915881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1422eacd-1572-41fc-94e4-e9269029e4a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:15.916157 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1422eacd-1572-41fc-94e4-e9269029e4a0 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:15.924589 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1422eacd-1572-41fc-94e4-e9269029e4a0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:15.926685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1422eacd-1572-41fc-94e4-e9269029e4a0 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:15.928541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d451ee-83e4-4579-aa6d-d332ccb3e645 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:15.940180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-1422eacd-1572-41fc-94e4-e9269029e4a0 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 04 00:47:15.940180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:15.940180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:15.940180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:15.940180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:15.940180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:15.940180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:15.940180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:15.940180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:15.940180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:15.940180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:15.940180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:15.940180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:15.940180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:15.940180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:15.940180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:15.941197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:15.942152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:15.942152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:15.942152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:15.942152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:15.942152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:15.942152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:15.942152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:15.942152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:15.942152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 04 00:47:15.942152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:15.942152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 314/631] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:47:15 2022] GET /identity/v3/users/glance => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:15.943119 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f0f829-c377-4495-965d-9faa778d5b42 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:15.943283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f0f829-c377-4495-965d-9faa778d5b42 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:15.947300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f0f829-c377-4495-965d-9faa778d5b42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:15.964505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-51f0f829-c377-4495-965d-9faa778d5b42 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:15.965928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-51f0f829-c377-4495-965d-9faa778d5b42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:15.966305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953f2379-b65c-489a-b78e-016c428c51ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:15.966909 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953f2379-b65c-489a-b78e-016c428c51ab None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:15.967187 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953f2379-b65c-489a-b78e-016c428c51ab None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:15.967459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953f2379-b65c-489a-b78e-016c428c51ab None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:15.968124 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953f2379-b65c-489a-b78e-016c428c51ab None admin] RBAC: Adding query filter params (name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:15.968427 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953f2379-b65c-489a-b78e-016c428c51ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:15.969809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953f2379-b65c-489a-b78e-016c428c51ab None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:15.981238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 318/632] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:15 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:15.981238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04690d3a-a2dc-439d-9e4f-df59bdb05422 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:15.981238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04690d3a-a2dc-439d-9e4f-df59bdb05422 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:15.981238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04690d3a-a2dc-439d-9e4f-df59bdb05422 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:15.985211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 319/633] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:15.985211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04690d3a-a2dc-439d-9e4f-df59bdb05422 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:16.053147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 315/634] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:16.053147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f0f829-c377-4495-965d-9faa778d5b42 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:16.109632 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04690d3a-a2dc-439d-9e4f-df59bdb05422 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:16.158712 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6737a8-ff07-4b24-a7ce-f8ac81ad8280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:16.159691 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6737a8-ff07-4b24-a7ce-f8ac81ad8280 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:16.160218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6737a8-ff07-4b24-a7ce-f8ac81ad8280 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:16.160897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6737a8-ff07-4b24-a7ce-f8ac81ad8280 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:16.161872 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6737a8-ff07-4b24-a7ce-f8ac81ad8280 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:16.162896 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6737a8-ff07-4b24-a7ce-f8ac81ad8280 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:16.164588 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6737a8-ff07-4b24-a7ce-f8ac81ad8280 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:16.168821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:16.194501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 320/635] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Sep 4 00:47:15 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:16.194501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b2d485-ebca-43f1-8d9f-4f5beabc6169 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:16.194501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b2d485-ebca-43f1-8d9f-4f5beabc6169 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:16.194501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b2d485-ebca-43f1-8d9f-4f5beabc6169 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:16.195744 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:16.204722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f0f829-c377-4495-965d-9faa778d5b42 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:16.214648 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-71b2d485-ebca-43f1-8d9f-4f5beabc6169 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:16.216115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-71b2d485-ebca-43f1-8d9f-4f5beabc6169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:16.249603 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01738463-c1d2-471a-8734-a3480afe154e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:16.250526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01738463-c1d2-471a-8734-a3480afe154e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:16.250813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01738463-c1d2-471a-8734-a3480afe154e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:16.251093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01738463-c1d2-471a-8734-a3480afe154e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:16.263821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01738463-c1d2-471a-8734-a3480afe154e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:16.265765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01738463-c1d2-471a-8734-a3480afe154e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:16.286239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-01738463-c1d2-471a-8734-a3480afe154e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:16.286239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:16.286239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:16.286239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:16.286239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:16.286239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:16.286239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:16.286239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:16.286239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:16.286239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:16.286239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:16.286239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:16.286239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:16.286239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:16.286239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:16.286239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:16.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:16.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:16.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:16.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:16.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:16.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:16.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:16.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:16.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:16.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:16.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:16.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:16.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:16.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:16.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:16.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:16.287237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:16.288134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:16.288134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:16.288134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:16.293203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 316/636] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:16.294223 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01738463-c1d2-471a-8734-a3480afe154e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:16.308888 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 321/637] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:16.308888 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b2d485-ebca-43f1-8d9f-4f5beabc6169 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:16.426867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b2d485-ebca-43f1-8d9f-4f5beabc6169 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:16.449738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01738463-c1d2-471a-8734-a3480afe154e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:16.462610 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:16.473600 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84c8b53-b4fa-4fe7-bdd0-238f0a46e335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:16.475690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84c8b53-b4fa-4fe7-bdd0-238f0a46e335 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:16.475690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84c8b53-b4fa-4fe7-bdd0-238f0a46e335 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:16.475690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84c8b53-b4fa-4fe7-bdd0-238f0a46e335 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:16.476186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84c8b53-b4fa-4fe7-bdd0-238f0a46e335 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:16.476603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84c8b53-b4fa-4fe7-bdd0-238f0a46e335 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:16.478211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84c8b53-b4fa-4fe7-bdd0-238f0a46e335 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:16.496896 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf48fb5-212e-44a5-a276-088c2b20bcfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:16.498161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf48fb5-212e-44a5-a276-088c2b20bcfc None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:16.498554 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf48fb5-212e-44a5-a276-088c2b20bcfc None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:16.499181 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 322/638] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:16.499605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf48fb5-212e-44a5-a276-088c2b20bcfc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:16.500258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84c8b53-b4fa-4fe7-bdd0-238f0a46e335 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:16.520002 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:16.577101 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf48fb5-212e-44a5-a276-088c2b20bcfc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:16.578868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf48fb5-212e-44a5-a276-088c2b20bcfc None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:16.633248 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84c8b53-b4fa-4fe7-bdd0-238f0a46e335 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:16.668267 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1e7944-6ddc-447f-b765-4fa38f3813fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:16.668893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1e7944-6ddc-447f-b765-4fa38f3813fc None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:16.669210 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1e7944-6ddc-447f-b765-4fa38f3813fc None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:16.669487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1e7944-6ddc-447f-b765-4fa38f3813fc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:16.679916 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1e7944-6ddc-447f-b765-4fa38f3813fc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:16.685242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-ab1e7944-6ddc-447f-b765-4fa38f3813fc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:16.685242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:16.685242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:16.685242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:16.685242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:16.685242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:16.685242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:16.685242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:16.685242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:16.685242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:16.685242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:16.685242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:16.685242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:16.685242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:16.685242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:16.685242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:16.686162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:16.686162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:16.686162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:16.686162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:16.686162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:16.686162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:16.686162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:16.686162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:16.686162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:16.686162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:16.686162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:16.686162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:16.686162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:16.686162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:16.686162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:16.686162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:16.687115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:16.687115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:16.687115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:16.687115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:16.687115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:16.687115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:16.687115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:16.687115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:16.687115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:16.687115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:16.687115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:16.687115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:16.687115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 323/639] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:16.687115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1e7944-6ddc-447f-b765-4fa38f3813fc None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:16.700526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 317/640] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:47:16.701944 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf48fb5-212e-44a5-a276-088c2b20bcfc None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:16.734600 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:16.758303 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:16.794271 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1e7944-6ddc-447f-b765-4fa38f3813fc None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:16.817480 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf48fb5-212e-44a5-a276-088c2b20bcfc None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:16.829161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08c9167-1f6f-4e30-a5c7-31d1c8f8ab51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:16.830230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08c9167-1f6f-4e30-a5c7-31d1c8f8ab51 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:16.830609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08c9167-1f6f-4e30-a5c7-31d1c8f8ab51 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:16.831438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08c9167-1f6f-4e30-a5c7-31d1c8f8ab51 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:16.840579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08c9167-1f6f-4e30-a5c7-31d1c8f8ab51 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:16.842330 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08c9167-1f6f-4e30-a5c7-31d1c8f8ab51 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:16.853975 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78b8de4-487d-4641-9935-4ad6b491c623 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:16.854603 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b8de4-487d-4641-9935-4ad6b491c623 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:16.854885 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b8de4-487d-4641-9935-4ad6b491c623 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:16.855153 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b8de4-487d-4641-9935-4ad6b491c623 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:16.855830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-c08c9167-1f6f-4e30-a5c7-31d1c8f8ab51 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:47:16.855830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:16.855830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:16.855830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:16.855830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:16.855830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:16.855830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:16.855830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:16.855830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:16.855830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:16.855830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:16.855830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:16.855830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:16.855830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:16.855830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:16.855830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:16.856724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:16.857706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:16.857706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:16.857706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:16.857706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:16.857706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:16.857706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:16.857706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:16.857706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:16.857706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:47:16.857706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:16.857706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 324/641] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 4 00:47:16 2022] GET /identity/v3/users/nova => generated 82 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:16.858300 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08c9167-1f6f-4e30-a5c7-31d1c8f8ab51 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:16.871212 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78b8de4-487d-4641-9935-4ad6b491c623 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:16.878077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-e78b8de4-487d-4641-9935-4ad6b491c623 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:16.878077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:16.878077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:16.878077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:16.878077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:16.878077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:16.878077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:16.878077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:16.878077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:16.878077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:16.878077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:16.878077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:16.878077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:16.878077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:16.878077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:16.878077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:16.879050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:16.879050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:16.879050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:16.879050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:16.879050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:16.879050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:16.879050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:16.879050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:16.879050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:16.879050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:16.879050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:16.879050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:16.879050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:16.879050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:16.879050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:16.879050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:16.880355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:16.880355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:16.880355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:16.880355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:16.880355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:16.880355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:16.880355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:16.880355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:16.880355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:16.880355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:16.880355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:16.880355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:16.880355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 318/642] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:16.880355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78b8de4-487d-4641-9935-4ad6b491c623 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:16.970034 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08c9167-1f6f-4e30-a5c7-31d1c8f8ab51 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:16.972824 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:16.989701 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78b8de4-487d-4641-9935-4ad6b491c623 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:16.999267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:17.007120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7ad15c-7f1c-4694-90c1-9e6ab99bd3cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:17.007737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7ad15c-7f1c-4694-90c1-9e6ab99bd3cf None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:17.008020 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7ad15c-7f1c-4694-90c1-9e6ab99bd3cf None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:17.008299 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7ad15c-7f1c-4694-90c1-9e6ab99bd3cf None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:17.008977 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7ad15c-7f1c-4694-90c1-9e6ab99bd3cf None admin] RBAC: Adding query filter params (name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:17.009315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7ad15c-7f1c-4694-90c1-9e6ab99bd3cf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:17.013794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7ad15c-7f1c-4694-90c1-9e6ab99bd3cf None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:17.019062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 325/643] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:17.019764 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7ad15c-7f1c-4694-90c1-9e6ab99bd3cf None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:17.028963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e3f96a-89b6-49e8-a293-e1c9a54dab11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:17.029798 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e3f96a-89b6-49e8-a293-e1c9a54dab11 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:17.030251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e3f96a-89b6-49e8-a293-e1c9a54dab11 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:17.030671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e3f96a-89b6-49e8-a293-e1c9a54dab11 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:17.044948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e3f96a-89b6-49e8-a293-e1c9a54dab11 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:17.047141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e3f96a-89b6-49e8-a293-e1c9a54dab11 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:17.062363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-c0e3f96a-89b6-49e8-a293-e1c9a54dab11 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:17.062363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:17.062363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:17.062363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:17.062363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:17.062363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:17.062363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:17.062363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:17.062363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:17.062363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:17.062363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:17.062363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:17.062363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:17.062363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:17.062363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:17.062363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:17.063300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:17.063300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:17.063300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:17.063300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:17.063300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:17.063300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:17.063300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:17.063300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:17.063300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:17.063300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:17.063300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:17.063300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:17.063300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:17.063300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:17.063300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:17.063300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:17.063300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:17.064347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:17.064347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:17.064347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:17.064347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 319/644] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:17.064827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e3f96a-89b6-49e8-a293-e1c9a54dab11 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:17.133810 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7ad15c-7f1c-4694-90c1-9e6ab99bd3cf None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:17.168016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7e2a1a-4dba-4091-b208-8fcf40a482ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:17.168624 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7e2a1a-4dba-4091-b208-8fcf40a482ed None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:17.168902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7e2a1a-4dba-4091-b208-8fcf40a482ed None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:17.169204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7e2a1a-4dba-4091-b208-8fcf40a482ed None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:17.169780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e3f96a-89b6-49e8-a293-e1c9a54dab11 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:17.169969 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7e2a1a-4dba-4091-b208-8fcf40a482ed None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:17.170237 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7e2a1a-4dba-4091-b208-8fcf40a482ed None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:17.172046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7e2a1a-4dba-4091-b208-8fcf40a482ed None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:17.195710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 326/645] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 4 00:47:17 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:17.196824 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7e2a1a-4dba-4091-b208-8fcf40a482ed None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:17.214392 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77343f39-06ce-449a-96c4-c9529f4de92d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:17.215930 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77343f39-06ce-449a-96c4-c9529f4de92d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:17.216211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77343f39-06ce-449a-96c4-c9529f4de92d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:17.216479 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77343f39-06ce-449a-96c4-c9529f4de92d None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:17.217169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77343f39-06ce-449a-96c4-c9529f4de92d None admin] RBAC: Adding query filter params (name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:17.217475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77343f39-06ce-449a-96c4-c9529f4de92d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:17.218830 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77343f39-06ce-449a-96c4-c9529f4de92d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:17.227445 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 320/646] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:17 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:17.228217 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77343f39-06ce-449a-96c4-c9529f4de92d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:17.230398 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:17.254407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:17.308397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7e2a1a-4dba-4091-b208-8fcf40a482ed None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:17.338683 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77343f39-06ce-449a-96c4-c9529f4de92d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:17.343031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89e563d-8bbe-4ead-8453-5f2be096aa08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:17.343669 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89e563d-8bbe-4ead-8453-5f2be096aa08 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:17.343947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89e563d-8bbe-4ead-8453-5f2be096aa08 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:17.344217 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89e563d-8bbe-4ead-8453-5f2be096aa08 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:17.354649 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89e563d-8bbe-4ead-8453-5f2be096aa08 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:17.356162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89e563d-8bbe-4ead-8453-5f2be096aa08 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:17.371163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-b89e563d-8bbe-4ead-8453-5f2be096aa08 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:17.371163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:17.371163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:17.371163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:17.371163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:17.371163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:17.371163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:17.371163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:17.371163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:17.371163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:17.371163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:17.371163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:17.371163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:17.371163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:17.371163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:17.371163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:17.372176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:17.372176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:17.372176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:17.372176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:17.372176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:17.372176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:17.372176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:17.372176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:17.372176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:17.372176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:17.372176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:17.372176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:17.372176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:17.372176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:17.372176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:17.372176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:17.372176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:17.373456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:17.373456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:17.373456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:17.373456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 327/647] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:17.373456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89e563d-8bbe-4ead-8453-5f2be096aa08 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:17.381044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33904391-bcdc-45df-ad71-119b5c466ae0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:17.381932 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33904391-bcdc-45df-ad71-119b5c466ae0 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:17.382452 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33904391-bcdc-45df-ad71-119b5c466ae0 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:17.382930 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33904391-bcdc-45df-ad71-119b5c466ae0 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:17.383851 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33904391-bcdc-45df-ad71-119b5c466ae0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:17.384227 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33904391-bcdc-45df-ad71-119b5c466ae0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:17.385608 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33904391-bcdc-45df-ad71-119b5c466ae0 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:17.401340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 321/648] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:17.401340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33904391-bcdc-45df-ad71-119b5c466ae0 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:17.483609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89e563d-8bbe-4ead-8453-5f2be096aa08 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:17.510221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33904391-bcdc-45df-ad71-119b5c466ae0 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:17.519040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248bd93b-f519-4d66-88d3-1fd81f0dc283 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:17.519650 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248bd93b-f519-4d66-88d3-1fd81f0dc283 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:17.519929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248bd93b-f519-4d66-88d3-1fd81f0dc283 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:17.520203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248bd93b-f519-4d66-88d3-1fd81f0dc283 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:17.537950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:17.545853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d04146-3906-48b6-8cbe-cb978f024f3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:17.546444 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d04146-3906-48b6-8cbe-cb978f024f3a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:17.546840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d04146-3906-48b6-8cbe-cb978f024f3a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:17.547121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d04146-3906-48b6-8cbe-cb978f024f3a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:17.559934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d04146-3906-48b6-8cbe-cb978f024f3a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:17.562742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d04146-3906-48b6-8cbe-cb978f024f3a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:17.582964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-73d04146-3906-48b6-8cbe-cb978f024f3a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:17.582964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:17.582964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:17.582964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:17.582964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:17.582964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:17.582964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:17.582964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:17.582964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:17.582964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:17.582964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:17.582964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:17.582964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:17.582964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:17.582964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:17.582964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:17.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:17.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:17.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:17.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:17.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:17.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:17.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:17.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:17.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:17.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:17.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:17.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:17.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:17.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:17.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:17.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:17.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:17.587746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:17.587746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:17.587746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:17.588434 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:17.589423 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 322/649] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:17.590554 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d04146-3906-48b6-8cbe-cb978f024f3a None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:17.593681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248bd93b-f519-4d66-88d3-1fd81f0dc283 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:17.595328 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248bd93b-f519-4d66-88d3-1fd81f0dc283 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:17.677024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:17.700996 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 328/650] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:47:17.702043 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248bd93b-f519-4d66-88d3-1fd81f0dc283 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:17.710571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d04146-3906-48b6-8cbe-cb978f024f3a None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:17.743644 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba18fe7c-76c9-43f0-b24a-48af1f96554f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:17.744188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba18fe7c-76c9-43f0-b24a-48af1f96554f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:17.744509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba18fe7c-76c9-43f0-b24a-48af1f96554f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:17.744811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba18fe7c-76c9-43f0-b24a-48af1f96554f None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:17.745515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba18fe7c-76c9-43f0-b24a-48af1f96554f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:17.745852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba18fe7c-76c9-43f0-b24a-48af1f96554f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:17.747265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba18fe7c-76c9-43f0-b24a-48af1f96554f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:17.748924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:17.761811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 323/651] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:17.763041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba18fe7c-76c9-43f0-b24a-48af1f96554f None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:17.803477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248bd93b-f519-4d66-88d3-1fd81f0dc283 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:17.835985 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b9ff22-8a89-4842-8f3f-6ca75b3656a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:17.836614 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b9ff22-8a89-4842-8f3f-6ca75b3656a5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:17.836894 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b9ff22-8a89-4842-8f3f-6ca75b3656a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:17.837204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b9ff22-8a89-4842-8f3f-6ca75b3656a5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:17.843574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b9ff22-8a89-4842-8f3f-6ca75b3656a5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:17.845020 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b9ff22-8a89-4842-8f3f-6ca75b3656a5 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:17.857354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-04b9ff22-8a89-4842-8f3f-6ca75b3656a5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:17.857354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:17.857354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:17.857354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:17.857354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:17.857354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:17.857354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:17.857354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:17.857354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:17.857354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:17.857354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:17.857354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:17.857354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:17.857354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:17.857354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:17.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:17.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:17.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:17.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:17.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:17.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:17.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:17.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:17.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:17.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:17.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:17.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:17.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:17.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:17.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:17.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:17.858404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:17.859480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:17.859480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:17.859480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:17.859480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:17.859480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:17.859480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:17.859480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:17.859480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:17.859480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 329/652] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:17.860096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950e439d-2fc6-4988-a5ab-6231e763463f None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:17.860096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950e439d-2fc6-4988-a5ab-6231e763463f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:17.860096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950e439d-2fc6-4988-a5ab-6231e763463f None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:17.861528 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 330/653] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:17 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:17.862881 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950e439d-2fc6-4988-a5ab-6231e763463f None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:17.869176 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba18fe7c-76c9-43f0-b24a-48af1f96554f None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:17.899962 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:17.904108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25623949-b89a-464d-b0b4-9f15f249aee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:17.904729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25623949-b89a-464d-b0b4-9f15f249aee9 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:17.905047 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25623949-b89a-464d-b0b4-9f15f249aee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:17.905385 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25623949-b89a-464d-b0b4-9f15f249aee9 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:17.910664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25623949-b89a-464d-b0b4-9f15f249aee9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:17.910664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25623949-b89a-464d-b0b4-9f15f249aee9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:17.910664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25623949-b89a-464d-b0b4-9f15f249aee9 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:17.919612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 324/654] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:17.920434 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25623949-b89a-464d-b0b4-9f15f249aee9 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:17.968723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950e439d-2fc6-4988-a5ab-6231e763463f None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:17.975359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:18.002595 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf88dbd-d6a1-4020-914c-f8c7135317c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:18.003352 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf88dbd-d6a1-4020-914c-f8c7135317c3 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:18.003559 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf88dbd-d6a1-4020-914c-f8c7135317c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:18.003838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf88dbd-d6a1-4020-914c-f8c7135317c3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:18.013624 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf88dbd-d6a1-4020-914c-f8c7135317c3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:18.015081 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf88dbd-d6a1-4020-914c-f8c7135317c3 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:18.024644 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25623949-b89a-464d-b0b4-9f15f249aee9 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:18.029920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-baf88dbd-d6a1-4020-914c-f8c7135317c3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:18.029920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:18.029920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:18.029920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:18.029920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:18.029920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:18.029920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:18.029920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:18.029920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:18.029920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:18.029920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:18.029920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:18.029920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:18.029920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:18.029920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:18.029920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:18.031004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:18.031004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:18.031004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:18.031004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:18.031004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:18.031004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:18.031004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:18.031004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:18.031004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:18.031004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:18.031004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:18.031004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:18.031004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:18.031004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:18.031004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:18.031004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:18.031004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:18.032117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:18.032117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:18.032117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:18.032117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 331/655] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:18.032117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf88dbd-d6a1-4020-914c-f8c7135317c3 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:18.060680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2c7cc4-397e-4aa4-8c50-8949665162ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:18.061351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2c7cc4-397e-4aa4-8c50-8949665162ec None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:18.061670 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2c7cc4-397e-4aa4-8c50-8949665162ec None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:18.061983 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2c7cc4-397e-4aa4-8c50-8949665162ec None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:18.069485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db2c7cc4-397e-4aa4-8c50-8949665162ec None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:18.070946 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db2c7cc4-397e-4aa4-8c50-8949665162ec None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:18.082867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-db2c7cc4-397e-4aa4-8c50-8949665162ec None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 04 00:47:18.082867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:18.082867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:18.082867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:18.082867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:18.082867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:18.082867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:18.082867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:18.082867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:18.082867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:18.082867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:18.082867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:18.082867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:18.082867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:18.082867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:18.082867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:18.083840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:18.084835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:18.084835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:18.084835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:18.084835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:18.084835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:18.084835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:18.084835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:18.084835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:18.084835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 04 00:47:18.084835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:18.084835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 325/656] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:17 2022] GET /identity/v3/users/neutron => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:18.085485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a401b2-2c1a-4387-8488-707b8e85bd71 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:18.085485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a401b2-2c1a-4387-8488-707b8e85bd71 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:18.085724 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a401b2-2c1a-4387-8488-707b8e85bd71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:18.103892 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-f8a401b2-2c1a-4387-8488-707b8e85bd71 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:18.105686 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f8a401b2-2c1a-4387-8488-707b8e85bd71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:18.121947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:18.133980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf88dbd-d6a1-4020-914c-f8c7135317c3 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:18.168898 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4dd620-1b93-4a7c-8f0c-cabbd7a82b84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:18.169524 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4dd620-1b93-4a7c-8f0c-cabbd7a82b84 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:18.169802 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4dd620-1b93-4a7c-8f0c-cabbd7a82b84 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:18.170104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4dd620-1b93-4a7c-8f0c-cabbd7a82b84 None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:18.170758 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4dd620-1b93-4a7c-8f0c-cabbd7a82b84 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:18.171066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4dd620-1b93-4a7c-8f0c-cabbd7a82b84 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:18.172857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4dd620-1b93-4a7c-8f0c-cabbd7a82b84 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:18.175242 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 326/657] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:18.176248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a401b2-2c1a-4387-8488-707b8e85bd71 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:18.185147 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 332/658] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:18.186004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4dd620-1b93-4a7c-8f0c-cabbd7a82b84 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:18.228922 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:18.288177 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4dd620-1b93-4a7c-8f0c-cabbd7a82b84 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:18.289998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a401b2-2c1a-4387-8488-707b8e85bd71 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:18.322170 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbb1c79-0e1c-4440-8f33-e4326e4d506b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:18.322773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbb1c79-0e1c-4440-8f33-e4326e4d506b None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:18.323054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbb1c79-0e1c-4440-8f33-e4326e4d506b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:18.323328 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbb1c79-0e1c-4440-8f33-e4326e4d506b None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:18.323964 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbb1c79-0e1c-4440-8f33-e4326e4d506b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:18.324285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbb1c79-0e1c-4440-8f33-e4326e4d506b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:18.325685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbb1c79-0e1c-4440-8f33-e4326e4d506b None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:18.326957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911c2997-5695-4096-a3e3-5d3cbd6ee5ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:18.327655 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911c2997-5695-4096-a3e3-5d3cbd6ee5ec None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:18.327935 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911c2997-5695-4096-a3e3-5d3cbd6ee5ec None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:18.328204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911c2997-5695-4096-a3e3-5d3cbd6ee5ec None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:18.336606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911c2997-5695-4096-a3e3-5d3cbd6ee5ec None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:18.338038 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911c2997-5695-4096-a3e3-5d3cbd6ee5ec None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:18.342826 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 333/659] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:18.342826 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbb1c79-0e1c-4440-8f33-e4326e4d506b None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:18.351487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:18.354708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-911c2997-5695-4096-a3e3-5d3cbd6ee5ec None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 04 00:47:18.354708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:18.354708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:18.354708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:18.354708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:18.354708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:18.354708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:18.354708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:18.354708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:18.354708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:18.354708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:18.354708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:18.354708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:18.354708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:18.354708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:18.354708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:18.355621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:18.357404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:18.357404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:18.357404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:18.357404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:18.357404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:18.357404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:18.357404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:18.357404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:18.357404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 04 00:47:18.357404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:18.357404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 327/660] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:47:18 2022] GET /identity/v3/users/cinder => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:18.357404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911c2997-5695-4096-a3e3-5d3cbd6ee5ec None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:18.453236 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbb1c79-0e1c-4440-8f33-e4326e4d506b None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:18.465026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911c2997-5695-4096-a3e3-5d3cbd6ee5ec None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:18.467343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:18.488719 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbd7798-6668-4324-82e3-bf4e7a906a81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:18.489475 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbd7798-6668-4324-82e3-bf4e7a906a81 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:18.489755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbd7798-6668-4324-82e3-bf4e7a906a81 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:18.490027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbd7798-6668-4324-82e3-bf4e7a906a81 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:18.490758 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bbd7798-6668-4324-82e3-bf4e7a906a81 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:18.491064 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bbd7798-6668-4324-82e3-bf4e7a906a81 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:18.492495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bbd7798-6668-4324-82e3-bf4e7a906a81 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:18.499801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e9b41f-f675-48b4-96ab-65ad5ea91591 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:18.500395 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e9b41f-f675-48b4-96ab-65ad5ea91591 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:18.500670 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e9b41f-f675-48b4-96ab-65ad5ea91591 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:18.500939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e9b41f-f675-48b4-96ab-65ad5ea91591 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:18.501683 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e9b41f-f675-48b4-96ab-65ad5ea91591 None admin] RBAC: Adding query filter params (role.id=df912a77fbaa479db172f76a214f8bc8, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=69153344fc4b45588bdbcecdfcfd7c3f) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:18.501989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e9b41f-f675-48b4-96ab-65ad5ea91591 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:18.503596 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e9b41f-f675-48b4-96ab-65ad5ea91591 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:18.509386 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 328/661] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:18 2022] GET /identity/v3/role_assignments?user.id=69153344fc4b45588bdbcecdfcfd7c3f&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=df912a77fbaa479db172f76a214f8bc8 => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:18.512666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 334/662] 10.222.0.42 () {56 vars in 1139 bytes} [Sun Sep 4 00:47:18 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:18.515115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbd7798-6668-4324-82e3-bf4e7a906a81 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:18.515331 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e9b41f-f675-48b4-96ab-65ad5ea91591 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:18.604438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:18.636722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e9b41f-f675-48b4-96ab-65ad5ea91591 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:18.644711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbd7798-6668-4324-82e3-bf4e7a906a81 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:18.676328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400e0856-2908-427f-a05d-98ed511f20be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:18.677243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400e0856-2908-427f-a05d-98ed511f20be None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:18.677530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400e0856-2908-427f-a05d-98ed511f20be None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:18.677795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400e0856-2908-427f-a05d-98ed511f20be None admin] PATH_INFO: `/v3/projects/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:18.687350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400e0856-2908-427f-a05d-98ed511f20be None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:18.687665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400e0856-2908-427f-a05d-98ed511f20be None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:18.694846 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1865ee0c-fedd-4f99-9a17-d7c1e0b2a303 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:18.695460 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1865ee0c-fedd-4f99-9a17-d7c1e0b2a303 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:18.695738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1865ee0c-fedd-4f99-9a17-d7c1e0b2a303 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:18.696007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1865ee0c-fedd-4f99-9a17-d7c1e0b2a303 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:18.696887 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1865ee0c-fedd-4f99-9a17-d7c1e0b2a303 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:18.697067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1865ee0c-fedd-4f99-9a17-d7c1e0b2a303 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:18.698490 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1865ee0c-fedd-4f99-9a17-d7c1e0b2a303 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:18.699854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-400e0856-2908-427f-a05d-98ed511f20be None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:18.699854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:18.699854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:18.699854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:18.699854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:18.699854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:18.699854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:18.699854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:18.699854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:18.699854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:18.699854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:18.699854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:18.699854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:18.699854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:18.699854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:18.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:18.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:18.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:18.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:18.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:18.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:18.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:18.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:18.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:18.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:18.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:18.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:18.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:18.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:18.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:18.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:18.700684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:18.701891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:18.701891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:18.701891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:18.701891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:18.701891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:18.701891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:18.701891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:18.701891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:18.702565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 329/663] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:18.703719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400e0856-2908-427f-a05d-98ed511f20be None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:18.725507 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 335/664] 10.222.0.42 () {56 vars in 1137 bytes} [Sun Sep 4 00:47:18 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:18.726807 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1865ee0c-fedd-4f99-9a17-d7c1e0b2a303 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:18.735523 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:18.810874 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400e0856-2908-427f-a05d-98ed511f20be None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:18.844526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1865ee0c-fedd-4f99-9a17-d7c1e0b2a303 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:18.848366 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc9ab85-6540-4787-bada-9f0c0ea2bcfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:18.849315 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc9ab85-6540-4787-bada-9f0c0ea2bcfc None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:18.849731 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc9ab85-6540-4787-bada-9f0c0ea2bcfc None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:18.850134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc9ab85-6540-4787-bada-9f0c0ea2bcfc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:18.865863 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc9ab85-6540-4787-bada-9f0c0ea2bcfc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:18.871679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-4fc9ab85-6540-4787-bada-9f0c0ea2bcfc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:18.871679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:18.871679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:18.871679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:18.871679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:18.871679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:18.871679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:18.871679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:18.871679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:18.871679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:18.871679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:18.871679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:18.871679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:18.871679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:18.871679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:18.871679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:18.872710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:18.872710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:18.872710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:18.872710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:18.872710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:18.872710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:18.872710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:18.872710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:18.872710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:18.872710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:18.872710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:18.872710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:18.872710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:18.872710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:18.872710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:18.872710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:18.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:18.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:18.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:18.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:18.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:18.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:18.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:18.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:18.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:18.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:18.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:18.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:18.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 330/665] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:18 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:18.874544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc9ab85-6540-4787-bada-9f0c0ea2bcfc None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:18.887356 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:18.899733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a156e75-fe2c-4124-89f3-67bd98befd7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:18.900339 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a156e75-fe2c-4124-89f3-67bd98befd7c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:18.900665 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a156e75-fe2c-4124-89f3-67bd98befd7c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:18.900940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a156e75-fe2c-4124-89f3-67bd98befd7c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:18.974525 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:18.980213 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a156e75-fe2c-4124-89f3-67bd98befd7c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:18.981688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a156e75-fe2c-4124-89f3-67bd98befd7c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:18.988169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc9ab85-6540-4787-bada-9f0c0ea2bcfc None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:19.025675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cecfeea-8843-43b2-9603-a01e4418ce21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:19.026283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cecfeea-8843-43b2-9603-a01e4418ce21 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:19.026588 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cecfeea-8843-43b2-9603-a01e4418ce21 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:19.026827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cecfeea-8843-43b2-9603-a01e4418ce21 None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:19.027517 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cecfeea-8843-43b2-9603-a01e4418ce21 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:19.027820 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cecfeea-8843-43b2-9603-a01e4418ce21 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:19.029280 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cecfeea-8843-43b2-9603-a01e4418ce21 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:19.042034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 331/666] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:19.043129 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cecfeea-8843-43b2-9603-a01e4418ce21 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:19.108436 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 336/667] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:47:19.108436 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a156e75-fe2c-4124-89f3-67bd98befd7c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:19.140988 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:19.152267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cecfeea-8843-43b2-9603-a01e4418ce21 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:19.194756 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf09c9f-88fb-4fdc-96f7-c3ff87695222 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:19.195372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf09c9f-88fb-4fdc-96f7-c3ff87695222 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:19.195649 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf09c9f-88fb-4fdc-96f7-c3ff87695222 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:19.195913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf09c9f-88fb-4fdc-96f7-c3ff87695222 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:19.214623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:19.234501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a156e75-fe2c-4124-89f3-67bd98befd7c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:19.270910 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf09c9f-88fb-4fdc-96f7-c3ff87695222 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:19.272697 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf09c9f-88fb-4fdc-96f7-c3ff87695222 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:19.276564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadb067d-cc69-478c-9366-2665451a3b9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:19.277393 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadb067d-cc69-478c-9366-2665451a3b9c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:19.277835 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadb067d-cc69-478c-9366-2665451a3b9c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:19.278266 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadb067d-cc69-478c-9366-2665451a3b9c None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:19.279502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadb067d-cc69-478c-9366-2665451a3b9c None admin] RBAC: Adding query filter params (name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:19.279986 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadb067d-cc69-478c-9366-2665451a3b9c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:19.281828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadb067d-cc69-478c-9366-2665451a3b9c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:19.293174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 337/668] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:19.293174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadb067d-cc69-478c-9366-2665451a3b9c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:19.384765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:19.388201 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 332/669] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:47:19.388831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf09c9f-88fb-4fdc-96f7-c3ff87695222 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:19.407594 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadb067d-cc69-478c-9366-2665451a3b9c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:19.440337 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49189b8-ca57-40c9-ac08-3596f6d69574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:19.440974 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49189b8-ca57-40c9-ac08-3596f6d69574 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:19.441280 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49189b8-ca57-40c9-ac08-3596f6d69574 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:19.441552 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49189b8-ca57-40c9-ac08-3596f6d69574 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:19.442272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49189b8-ca57-40c9-ac08-3596f6d69574 None admin] RBAC: Adding query filter params (role.id=df912a77fbaa479db172f76a214f8bc8, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=57c88f16c7df40adb917f49804fda21f) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:19.442581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49189b8-ca57-40c9-ac08-3596f6d69574 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:19.443995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49189b8-ca57-40c9-ac08-3596f6d69574 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:19.444117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:19.449814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 338/670] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:19 2022] GET /identity/v3/role_assignments?user.id=57c88f16c7df40adb917f49804fda21f&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=df912a77fbaa479db172f76a214f8bc8 => generated 608 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:19.453327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49189b8-ca57-40c9-ac08-3596f6d69574 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:19.516165 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf09c9f-88fb-4fdc-96f7-c3ff87695222 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:19.560557 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3e80ad-9c91-4a9f-93f6-cfee6415c2ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:19.561907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3e80ad-9c91-4a9f-93f6-cfee6415c2ed None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:19.562453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3e80ad-9c91-4a9f-93f6-cfee6415c2ed None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:19.563583 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3e80ad-9c91-4a9f-93f6-cfee6415c2ed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:19.579886 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3e80ad-9c91-4a9f-93f6-cfee6415c2ed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:19.583271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3e80ad-9c91-4a9f-93f6-cfee6415c2ed None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:19.605195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-2e3e80ad-9c91-4a9f-93f6-cfee6415c2ed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:19.605195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:19.605195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:19.605195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:19.605195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:19.605195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:19.605195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:19.605195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:19.605195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:19.605195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:19.605195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:19.605195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:19.605195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:19.605195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:19.605195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:19.605195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:19.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:19.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:19.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:19.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:19.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:19.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:19.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:19.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:19.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:19.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:19.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:19.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:19.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:19.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:19.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:19.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:19.606195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:19.607077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:19.607077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:19.607077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:19.607077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 333/671] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:19.607077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3e80ad-9c91-4a9f-93f6-cfee6415c2ed None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:19.611009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49189b8-ca57-40c9-ac08-3596f6d69574 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:19.657851 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada1e170-daaf-411a-b00d-e7a6067f5c39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:19.658412 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada1e170-daaf-411a-b00d-e7a6067f5c39 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:19.658694 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada1e170-daaf-411a-b00d-e7a6067f5c39 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:19.659524 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada1e170-daaf-411a-b00d-e7a6067f5c39 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:19.671261 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada1e170-daaf-411a-b00d-e7a6067f5c39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:19.672712 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada1e170-daaf-411a-b00d-e7a6067f5c39 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:19.689966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-ada1e170-daaf-411a-b00d-e7a6067f5c39 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:19.689966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:19.689966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:19.689966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:19.689966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:19.689966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:19.689966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:19.689966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:19.689966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:19.689966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:19.689966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:19.689966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:19.689966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:19.689966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:19.689966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:19.689966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:19.691026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:19.691026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:19.691026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:19.691026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:19.691026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:19.691026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:19.691026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:19.691026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:19.691026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:19.691026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:19.691026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:19.691026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:19.691026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:19.691026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:19.691026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:19.691026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:19.691026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:19.691960 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:19.691960 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:19.691960 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:19.691960 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 339/672] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:19.692194 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada1e170-daaf-411a-b00d-e7a6067f5c39 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:19.695198 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:19.744146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:19.747782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3e80ad-9c91-4a9f-93f6-cfee6415c2ed None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:19.790511 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78372340-4dbf-4046-8f0b-cfe9452ad662 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:19.791455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78372340-4dbf-4046-8f0b-cfe9452ad662 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:19.791943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78372340-4dbf-4046-8f0b-cfe9452ad662 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:19.792396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78372340-4dbf-4046-8f0b-cfe9452ad662 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:19.804045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78372340-4dbf-4046-8f0b-cfe9452ad662 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:19.805951 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78372340-4dbf-4046-8f0b-cfe9452ad662 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:19.821660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-78372340-4dbf-4046-8f0b-cfe9452ad662 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:19.821660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:19.821660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:19.821660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:19.821660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:19.821660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:19.821660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:19.821660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:19.821660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:19.821660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:19.821660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:19.821660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:19.821660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:19.821660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:19.821660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:19.821660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:19.823024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:19.823024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:19.823024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:19.823024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:19.823024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:19.823024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:19.823024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:19.823024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:19.823024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:19.823024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:19.823024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:19.823024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:19.823024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:19.823024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:19.823024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:19.823024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:19.823024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:19.823908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:19.823908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:19.823908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:19.826105 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 334/673] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:19.827185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78372340-4dbf-4046-8f0b-cfe9452ad662 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:19.832493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada1e170-daaf-411a-b00d-e7a6067f5c39 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:19.875058 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c47fb9c-f636-4387-9e3f-fa188531762a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:19.875621 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c47fb9c-f636-4387-9e3f-fa188531762a None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:19.875974 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c47fb9c-f636-4387-9e3f-fa188531762a None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:19.876321 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c47fb9c-f636-4387-9e3f-fa188531762a None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:19.877173 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c47fb9c-f636-4387-9e3f-fa188531762a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:19.878021 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c47fb9c-f636-4387-9e3f-fa188531762a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:19.879665 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c47fb9c-f636-4387-9e3f-fa188531762a None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:19.892064 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 340/674] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:19.892996 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c47fb9c-f636-4387-9e3f-fa188531762a None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:19.943042 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78372340-4dbf-4046-8f0b-cfe9452ad662 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:19.956891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:19.983175 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3915400b-87a0-4fee-b446-0b21f8a0e90a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:19.983971 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3915400b-87a0-4fee-b446-0b21f8a0e90a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:19.984251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3915400b-87a0-4fee-b446-0b21f8a0e90a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:19.984531 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3915400b-87a0-4fee-b446-0b21f8a0e90a None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:19.985268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3915400b-87a0-4fee-b446-0b21f8a0e90a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:19.985574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3915400b-87a0-4fee-b446-0b21f8a0e90a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:19.987436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3915400b-87a0-4fee-b446-0b21f8a0e90a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:20.001568 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c47fb9c-f636-4387-9e3f-fa188531762a None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:20.005154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 335/675] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:20.006286 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3915400b-87a0-4fee-b446-0b21f8a0e90a None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:20.030165 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:20.050891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d772dbd9-49ca-474d-8a97-5527d838a4bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:20.052047 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d772dbd9-49ca-474d-8a97-5527d838a4bc None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:20.052845 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d772dbd9-49ca-474d-8a97-5527d838a4bc None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:20.053685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d772dbd9-49ca-474d-8a97-5527d838a4bc None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:20.054616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d772dbd9-49ca-474d-8a97-5527d838a4bc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:20.055141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d772dbd9-49ca-474d-8a97-5527d838a4bc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:20.057747 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d772dbd9-49ca-474d-8a97-5527d838a4bc None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:20.078583 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 341/676] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:20.078583 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d772dbd9-49ca-474d-8a97-5527d838a4bc None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:20.152810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3915400b-87a0-4fee-b446-0b21f8a0e90a None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:20.209865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1236c36e-9689-486f-a1e4-2f9036dbf2de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:20.210851 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1236c36e-9689-486f-a1e4-2f9036dbf2de None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:20.211341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1236c36e-9689-486f-a1e4-2f9036dbf2de None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:20.211833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1236c36e-9689-486f-a1e4-2f9036dbf2de None admin] PATH_INFO: `/v3/projects/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:20.219980 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1236c36e-9689-486f-a1e4-2f9036dbf2de None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:20.221791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1236c36e-9689-486f-a1e4-2f9036dbf2de None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:20.226172 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d772dbd9-49ca-474d-8a97-5527d838a4bc None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:20.234976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-1236c36e-9689-486f-a1e4-2f9036dbf2de None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:20.234976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:20.234976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:20.234976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:20.234976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:20.234976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:20.234976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:20.234976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:20.234976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:20.234976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:20.234976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:20.234976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:20.234976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:20.234976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:20.234976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:20.236102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:20.236102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:20.236102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:20.236102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:20.236102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:20.236102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:20.236102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:20.236102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:20.236102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:20.236102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:20.236102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:20.236102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:20.236102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:20.236102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:20.236102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:20.236102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:20.236102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:20.237545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:20.237545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:20.237545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:20.237545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:20.237545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:20.237545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:20.237545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:20.237545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:20.240066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 336/677] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:20.241233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1236c36e-9689-486f-a1e4-2f9036dbf2de None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:20.251592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:20.263700 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbc88a5-ce02-4450-aa58-3f37a8ba9010 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:20.264376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbc88a5-ce02-4450-aa58-3f37a8ba9010 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:20.264610 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbc88a5-ce02-4450-aa58-3f37a8ba9010 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:20.264881 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbc88a5-ce02-4450-aa58-3f37a8ba9010 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:20.272722 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbc88a5-ce02-4450-aa58-3f37a8ba9010 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:20.274203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbc88a5-ce02-4450-aa58-3f37a8ba9010 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:20.286340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-cfbc88a5-ce02-4450-aa58-3f37a8ba9010 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 04 00:47:20.286340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:20.286340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:20.286340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:20.286340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:20.286340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:20.286340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:20.286340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:20.286340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:20.286340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:20.286340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:20.286340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:20.286340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:20.286340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:20.286340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:20.286340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:20.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:20.288241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:20.288241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:20.288241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:20.288241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:20.288241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:20.288241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:20.288241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:20.288241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:20.288241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 04 00:47:20.288241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:20.292355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 342/678] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:47:20 2022] GET /identity/v3/users/swift => generated 83 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:20.293244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbc88a5-ce02-4450-aa58-3f37a8ba9010 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:20.327702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:20.358465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1236c36e-9689-486f-a1e4-2f9036dbf2de None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:20.396519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472d7066-9241-44be-9153-0997ecacd96e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:20.397557 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472d7066-9241-44be-9153-0997ecacd96e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:20.398064 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472d7066-9241-44be-9153-0997ecacd96e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:20.398575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472d7066-9241-44be-9153-0997ecacd96e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:20.400193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbc88a5-ce02-4450-aa58-3f37a8ba9010 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:20.406427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472d7066-9241-44be-9153-0997ecacd96e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:20.408114 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472d7066-9241-44be-9153-0997ecacd96e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:20.420370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-472d7066-9241-44be-9153-0997ecacd96e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:20.420370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:20.420370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:20.420370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:20.420370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:20.420370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:20.420370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:20.420370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:20.420370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:20.420370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:20.420370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:20.420370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:20.420370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:20.420370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:20.420370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:20.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:20.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:20.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:20.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:20.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:20.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:20.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:20.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:20.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:20.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:20.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:20.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:20.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:20.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:20.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:20.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:20.421316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:20.422255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:20.422255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:20.422255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:20.422255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:20.422255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:20.422255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:20.422255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:20.422255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:20.430073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 337/679] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:20.431252 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472d7066-9241-44be-9153-0997ecacd96e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:20.436089 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c764f95-089a-4610-b48e-310b84ed93b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:20.436697 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c764f95-089a-4610-b48e-310b84ed93b0 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:20.436976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c764f95-089a-4610-b48e-310b84ed93b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:20.437286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c764f95-089a-4610-b48e-310b84ed93b0 None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:20.437982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c764f95-089a-4610-b48e-310b84ed93b0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:20.438285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c764f95-089a-4610-b48e-310b84ed93b0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:20.439721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c764f95-089a-4610-b48e-310b84ed93b0 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:20.452592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 343/680] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:20.453485 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c764f95-089a-4610-b48e-310b84ed93b0 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:20.498846 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:20.546741 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472d7066-9241-44be-9153-0997ecacd96e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:20.559811 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c764f95-089a-4610-b48e-310b84ed93b0 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:20.579864 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:20.585903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077efb8c-ef5a-4bcc-81b4-f36c798a086e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:20.586514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077efb8c-ef5a-4bcc-81b4-f36c798a086e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:20.586795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077efb8c-ef5a-4bcc-81b4-f36c798a086e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:20.587066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077efb8c-ef5a-4bcc-81b4-f36c798a086e None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:20.587795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077efb8c-ef5a-4bcc-81b4-f36c798a086e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:20.588101 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077efb8c-ef5a-4bcc-81b4-f36c798a086e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:20.589553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077efb8c-ef5a-4bcc-81b4-f36c798a086e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:20.598679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172141df-235d-4f48-a3e2-38e35f73f2ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:20.602741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172141df-235d-4f48-a3e2-38e35f73f2ae None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:20.602741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172141df-235d-4f48-a3e2-38e35f73f2ae None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:20.602741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172141df-235d-4f48-a3e2-38e35f73f2ae None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:20.602741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172141df-235d-4f48-a3e2-38e35f73f2ae None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:20.602741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172141df-235d-4f48-a3e2-38e35f73f2ae None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:20.602741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172141df-235d-4f48-a3e2-38e35f73f2ae None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:20.607122 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 338/681] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Sep 4 00:47:20 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:20.607934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077efb8c-ef5a-4bcc-81b4-f36c798a086e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:20.617118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 344/682] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:20.618453 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f65d99e-e64c-4849-8a35-ad9424af73f6 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:20.618738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f65d99e-e64c-4849-8a35-ad9424af73f6 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:20.619007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f65d99e-e64c-4849-8a35-ad9424af73f6 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:20.624979 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 345/683] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:20 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:20.625953 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f65d99e-e64c-4849-8a35-ad9424af73f6 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:20.714836 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077efb8c-ef5a-4bcc-81b4-f36c798a086e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:20.736982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f65d99e-e64c-4849-8a35-ad9424af73f6 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:20.738871 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:20.747673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2bbd00-a2ff-407c-b1c6-91186e0ec777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:20.751419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2bbd00-a2ff-407c-b1c6-91186e0ec777 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:20.751419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2bbd00-a2ff-407c-b1c6-91186e0ec777 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:20.751419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2bbd00-a2ff-407c-b1c6-91186e0ec777 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:20.751419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2bbd00-a2ff-407c-b1c6-91186e0ec777 None admin] RBAC: Adding query filter params (role.id=df912a77fbaa479db172f76a214f8bc8, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=da4f44b86ca94adf959d9a41d915c902) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:20.751419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2bbd00-a2ff-407c-b1c6-91186e0ec777 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:20.751780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2bbd00-a2ff-407c-b1c6-91186e0ec777 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:20.766284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 339/684] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:20 2022] GET /identity/v3/role_assignments?user.id=da4f44b86ca94adf959d9a41d915c902&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=df912a77fbaa479db172f76a214f8bc8 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:20.767573 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2bbd00-a2ff-407c-b1c6-91186e0ec777 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:20.781445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d9a0c5-8345-4cc7-99c0-39df8434d56c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:20.782287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d9a0c5-8345-4cc7-99c0-39df8434d56c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:20.783494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d9a0c5-8345-4cc7-99c0-39df8434d56c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:20.784451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d9a0c5-8345-4cc7-99c0-39df8434d56c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:20.857862 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:20.890191 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d9a0c5-8345-4cc7-99c0-39df8434d56c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:20.892259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d9a0c5-8345-4cc7-99c0-39df8434d56c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:20.919737 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2bbd00-a2ff-407c-b1c6-91186e0ec777 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:20.966771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88923eb-d3d0-406b-a6aa-1e2cf04d3ab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:20.967382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88923eb-d3d0-406b-a6aa-1e2cf04d3ab5 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:20.967668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88923eb-d3d0-406b-a6aa-1e2cf04d3ab5 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:20.967937 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88923eb-d3d0-406b-a6aa-1e2cf04d3ab5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:20.968667 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88923eb-d3d0-406b-a6aa-1e2cf04d3ab5 None admin] RBAC: Adding query filter params (role.id=df912a77fbaa479db172f76a214f8bc8, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=615a607b6be14b82979f94eabaf4c34c) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:20.968986 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88923eb-d3d0-406b-a6aa-1e2cf04d3ab5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:20.970410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88923eb-d3d0-406b-a6aa-1e2cf04d3ab5 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:20.978575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 340/685] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:20 2022] GET /identity/v3/role_assignments?user.id=615a607b6be14b82979f94eabaf4c34c&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=df912a77fbaa479db172f76a214f8bc8 => generated 608 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:20.980119 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ee409c-6401-47b6-95ea-0fb7cad6490b None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:20.980584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ee409c-6401-47b6-95ea-0fb7cad6490b None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:20.981035 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ee409c-6401-47b6-95ea-0fb7cad6490b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:21.001131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-25ee409c-6401-47b6-95ea-0fb7cad6490b None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:21.002260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-25ee409c-6401-47b6-95ea-0fb7cad6490b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:21.028042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 346/686] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:47:21.033320 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d9a0c5-8345-4cc7-99c0-39df8434d56c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:21.050012 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:21.099599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 341/687] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:21.212978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d9a0c5-8345-4cc7-99c0-39df8434d56c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:21.252109 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515e738f-c99e-452b-a568-be03b353e882 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:21.253547 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515e738f-c99e-452b-a568-be03b353e882 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:21.254011 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515e738f-c99e-452b-a568-be03b353e882 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:21.254486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515e738f-c99e-452b-a568-be03b353e882 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:21.265375 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515e738f-c99e-452b-a568-be03b353e882 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:21.267113 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515e738f-c99e-452b-a568-be03b353e882 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:21.286360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-515e738f-c99e-452b-a568-be03b353e882 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:21.286360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:21.286360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:21.286360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:21.286360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:21.286360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:21.286360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:21.286360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:21.286360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:21.286360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:21.286360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:21.286360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:21.286360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:21.286360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:21.286360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:21.286360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:21.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:21.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:21.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:21.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:21.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:21.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:21.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:21.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:21.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:21.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:21.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:21.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:21.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:21.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:21.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:21.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:21.287358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:21.288218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:21.288218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:21.288218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:21.290057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 347/688] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:21.297114 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ee409c-6401-47b6-95ea-0fb7cad6490b None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:21.371130 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:21.395918 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515e738f-c99e-452b-a568-be03b353e882 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:21.459103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ee409c-6401-47b6-95ea-0fb7cad6490b None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:21.490800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:21.517870 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd28bce-d658-42c4-99e3-287b7588f2e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:21.519078 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd28bce-d658-42c4-99e3-287b7588f2e0 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:21.519350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd28bce-d658-42c4-99e3-287b7588f2e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:21.519796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd28bce-d658-42c4-99e3-287b7588f2e0 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:21.520472 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd28bce-d658-42c4-99e3-287b7588f2e0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:21.520780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd28bce-d658-42c4-99e3-287b7588f2e0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:21.522321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd28bce-d658-42c4-99e3-287b7588f2e0 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:21.546919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515e738f-c99e-452b-a568-be03b353e882 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:21.547518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 342/689] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:21.553608 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd28bce-d658-42c4-99e3-287b7588f2e0 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:21.607142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27ee2f0-b6df-419d-899f-bbe67299bd0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:21.607785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27ee2f0-b6df-419d-899f-bbe67299bd0c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:21.608069 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27ee2f0-b6df-419d-899f-bbe67299bd0c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:21.608346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27ee2f0-b6df-419d-899f-bbe67299bd0c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:21.633333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27ee2f0-b6df-419d-899f-bbe67299bd0c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:21.634785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27ee2f0-b6df-419d-899f-bbe67299bd0c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:21.662373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-b27ee2f0-b6df-419d-899f-bbe67299bd0c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:21.662373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:21.662373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:21.662373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:21.662373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:21.662373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:21.662373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:21.662373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:21.662373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:21.662373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:21.662373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:21.662373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:21.662373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:21.662373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:21.662373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:21.662373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:21.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:21.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:21.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:21.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:21.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:21.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:21.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:21.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:21.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:21.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:21.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:21.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:21.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:21.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:21.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:21.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:21.663369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:21.664238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:21.664238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:21.664238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:21.664238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 348/690] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:21.673286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27ee2f0-b6df-419d-899f-bbe67299bd0c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:21.713645 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd28bce-d658-42c4-99e3-287b7588f2e0 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:21.724442 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:21.750784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d937c6-297b-4b56-8fbe-3e57ba01dbe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:21.751655 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d937c6-297b-4b56-8fbe-3e57ba01dbe3 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:21.752048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d937c6-297b-4b56-8fbe-3e57ba01dbe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:21.752454 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d937c6-297b-4b56-8fbe-3e57ba01dbe3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:21.760377 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d937c6-297b-4b56-8fbe-3e57ba01dbe3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:21.762075 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d937c6-297b-4b56-8fbe-3e57ba01dbe3 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:21.775354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-73d937c6-297b-4b56-8fbe-3e57ba01dbe3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:21.775354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:21.775354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:21.775354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:21.775354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:21.775354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:21.775354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:21.775354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:21.775354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:21.775354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:21.775354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:21.775354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:21.775354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:21.775354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:21.775354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:21.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:21.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:21.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:21.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:21.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:21.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:21.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:21.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:21.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:21.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:21.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:21.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:21.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:21.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:21.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:21.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:21.776437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:21.777494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:21.777494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:21.777494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:21.777494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:21.777494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:21.777494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:21.777494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:21.777494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:21.782605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 343/691] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:21.784390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a020ec-da52-49fb-bf5d-096be4095958 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:21.785256 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a020ec-da52-49fb-bf5d-096be4095958 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:21.785637 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a020ec-da52-49fb-bf5d-096be4095958 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:21.787369 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 344/692] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:21.788231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a020ec-da52-49fb-bf5d-096be4095958 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:21.789671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27ee2f0-b6df-419d-899f-bbe67299bd0c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:21.797604 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:21.827617 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df4a806-b3b8-4dec-8f81-cb20688e5d58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:21.828515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df4a806-b3b8-4dec-8f81-cb20688e5d58 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:21.829011 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df4a806-b3b8-4dec-8f81-cb20688e5d58 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:21.829490 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df4a806-b3b8-4dec-8f81-cb20688e5d58 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:21.830324 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df4a806-b3b8-4dec-8f81-cb20688e5d58 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:21.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df4a806-b3b8-4dec-8f81-cb20688e5d58 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:21.833216 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df4a806-b3b8-4dec-8f81-cb20688e5d58 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:21.846733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 349/693] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:21.848402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f92a68-be58-419a-8966-a80cf97adae0 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:21.848888 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f92a68-be58-419a-8966-a80cf97adae0 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:21.849403 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f92a68-be58-419a-8966-a80cf97adae0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:21.871650 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-a9f92a68-be58-419a-8966-a80cf97adae0 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:21.872856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a9f92a68-be58-419a-8966-a80cf97adae0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:21.904578 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a020ec-da52-49fb-bf5d-096be4095958 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:21.948958 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9300429a-8d24-448c-9f96-fac791a4aa47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:21.949925 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9300429a-8d24-448c-9f96-fac791a4aa47 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:21.950481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9300429a-8d24-448c-9f96-fac791a4aa47 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:21.950822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9300429a-8d24-448c-9f96-fac791a4aa47 None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:21.951597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9300429a-8d24-448c-9f96-fac791a4aa47 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:21.952016 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9300429a-8d24-448c-9f96-fac791a4aa47 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:21.953590 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9300429a-8d24-448c-9f96-fac791a4aa47 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:21.954443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 350/694] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:21.955403 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f92a68-be58-419a-8966-a80cf97adae0 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:21.984077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 345/695] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:21.994220 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9300429a-8d24-448c-9f96-fac791a4aa47 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:22.060226 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:22.081440 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:22.090319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f92a68-be58-419a-8966-a80cf97adae0 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:22.158250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9300429a-8d24-448c-9f96-fac791a4aa47 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:22.181781 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d243c161-6eae-434b-8311-eec4c13a85c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:22.182396 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d243c161-6eae-434b-8311-eec4c13a85c9 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:22.182673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d243c161-6eae-434b-8311-eec4c13a85c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:22.182946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d243c161-6eae-434b-8311-eec4c13a85c9 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:22.183596 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d243c161-6eae-434b-8311-eec4c13a85c9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:22.185038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d243c161-6eae-434b-8311-eec4c13a85c9 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:22.196767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8937c117-1274-4046-80b4-1d8741a39c29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:22.197409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8937c117-1274-4046-80b4-1d8741a39c29 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:22.197744 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8937c117-1274-4046-80b4-1d8741a39c29 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:22.198032 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8937c117-1274-4046-80b4-1d8741a39c29 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:22.199034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8937c117-1274-4046-80b4-1d8741a39c29 None admin] RBAC: Adding query filter params (role.id=df912a77fbaa479db172f76a214f8bc8, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=123ab6ff796c4e599ce1d6f4095bd898) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:22.199546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8937c117-1274-4046-80b4-1d8741a39c29 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:22.202351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8937c117-1274-4046-80b4-1d8741a39c29 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:22.212893 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 346/696] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:21 2022] GET /identity/v3/role_assignments?user.id=123ab6ff796c4e599ce1d6f4095bd898&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=df912a77fbaa479db172f76a214f8bc8 => generated 608 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:22.294692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8937c117-1274-4046-80b4-1d8741a39c29 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:22.329001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 351/697] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:47:21 2022] POST /identity/v3/users => generated 254 bytes in 374 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:47:22.416835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:22.433609 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8937c117-1274-4046-80b4-1d8741a39c29 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:22.480895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59e1954-1d42-46a9-8aab-9acc5e18bf9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:22.481586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59e1954-1d42-46a9-8aab-9acc5e18bf9f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:22.481871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59e1954-1d42-46a9-8aab-9acc5e18bf9f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:22.482146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59e1954-1d42-46a9-8aab-9acc5e18bf9f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:22.495803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59e1954-1d42-46a9-8aab-9acc5e18bf9f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:22.501465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-a59e1954-1d42-46a9-8aab-9acc5e18bf9f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:47:22.501465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:22.501465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:22.501465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:22.501465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:22.501465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:22.501465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:22.501465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:22.501465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:22.501465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:22.501465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:22.501465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:22.501465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:22.501465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:22.501465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:22.501465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:22.503705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:22.503705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:22.503705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:22.503705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:22.503705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:22.503705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:22.503705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:22.503705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:22.503705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:22.503705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:22.503705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:22.503705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:22.503705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:22.503705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:22.503705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:22.503705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:22.505675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:22.505675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:22.505675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:22.505675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:22.505675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:22.505675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:22.505675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:22.505675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:22.505675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:22.505675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:22.505675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:47:22.505675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:22.505675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 347/698] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:47:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:22.514289 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d243c161-6eae-434b-8311-eec4c13a85c9 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:22.636339 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d243c161-6eae-434b-8311-eec4c13a85c9 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:22.677591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ac5df3-745b-43ca-97f5-157b564a746c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:22.678214 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ac5df3-745b-43ca-97f5-157b564a746c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:22.678497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ac5df3-745b-43ca-97f5-157b564a746c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:22.678770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ac5df3-745b-43ca-97f5-157b564a746c None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:22.679444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ac5df3-745b-43ca-97f5-157b564a746c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:22.679744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ac5df3-745b-43ca-97f5-157b564a746c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:22.681138 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ac5df3-745b-43ca-97f5-157b564a746c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:22.690868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 352/699] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:47:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:22.696565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59e1954-1d42-46a9-8aab-9acc5e18bf9f None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:22.845366 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59e1954-1d42-46a9-8aab-9acc5e18bf9f None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:22.898527 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42eb785-cdc6-483e-bfed-e6f3eba63d37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:22.899108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42eb785-cdc6-483e-bfed-e6f3eba63d37 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:22.899390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42eb785-cdc6-483e-bfed-e6f3eba63d37 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:22.899663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42eb785-cdc6-483e-bfed-e6f3eba63d37 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:22.916513 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42eb785-cdc6-483e-bfed-e6f3eba63d37 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:22.917999 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42eb785-cdc6-483e-bfed-e6f3eba63d37 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:22.942516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-c42eb785-cdc6-483e-bfed-e6f3eba63d37 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:22.942516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:22.942516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:22.942516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:22.942516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:22.942516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:22.942516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:22.942516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:22.942516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:22.942516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:22.942516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:22.942516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:22.942516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:22.942516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:22.942516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:22.942516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:22.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:22.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:22.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:22.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:22.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:22.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:22.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:22.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:22.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:22.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:22.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:22.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:22.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:22.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:22.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:22.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:22.943521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:22.944402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:22.944402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:22.944402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:22.944402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 348/700] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:22.950786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ac5df3-745b-43ca-97f5-157b564a746c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:22.972519 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:23.066102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ac5df3-745b-43ca-97f5-157b564a746c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:23.102262 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03512a8b-786e-46d6-9e7b-8448c595aec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:23.102870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03512a8b-786e-46d6-9e7b-8448c595aec9 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:23.103150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03512a8b-786e-46d6-9e7b-8448c595aec9 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:23.103421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03512a8b-786e-46d6-9e7b-8448c595aec9 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:23.104070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03512a8b-786e-46d6-9e7b-8448c595aec9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:23.104375 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03512a8b-786e-46d6-9e7b-8448c595aec9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:23.105835 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03512a8b-786e-46d6-9e7b-8448c595aec9 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:23.117922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 353/701] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:23.126233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42eb785-cdc6-483e-bfed-e6f3eba63d37 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:23.130666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:23.142765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dec8c39-fcce-4125-a515-a9fc28c8a2a2 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:23.149140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dec8c39-fcce-4125-a515-a9fc28c8a2a2 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:23.149140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dec8c39-fcce-4125-a515-a9fc28c8a2a2 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:23.149140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 354/702] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:23.150087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1acf4a5-bb67-4125-82ac-f583cfcb00b8 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:23.150366 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1acf4a5-bb67-4125-82ac-f583cfcb00b8 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:23.150660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1acf4a5-bb67-4125-82ac-f583cfcb00b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:23.184591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-f1acf4a5-bb67-4125-82ac-f583cfcb00b8 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:23.186474 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-f1acf4a5-bb67-4125-82ac-f583cfcb00b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:23.288215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42eb785-cdc6-483e-bfed-e6f3eba63d37 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:23.297174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 355/703] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:47:23.340567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53220b88-19e3-4528-9e6f-bd89a8d9ac9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:23.341228 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53220b88-19e3-4528-9e6f-bd89a8d9ac9a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:23.341518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53220b88-19e3-4528-9e6f-bd89a8d9ac9a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:23.341791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53220b88-19e3-4528-9e6f-bd89a8d9ac9a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:23.352659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53220b88-19e3-4528-9e6f-bd89a8d9ac9a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:23.354310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53220b88-19e3-4528-9e6f-bd89a8d9ac9a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:23.373229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-53220b88-19e3-4528-9e6f-bd89a8d9ac9a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:47:23.373229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:23.373229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:23.373229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:23.373229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:23.373229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:23.373229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:23.373229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:23.373229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:23.373229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:23.373229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:23.373229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:23.373229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:23.373229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:23.373229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:23.373229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:23.374246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:23.375211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:23.375211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:23.375211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:23.375211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:23.375211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:23.375211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:23.375211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:23.375211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:23.375211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:47:23.375211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:23.379372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 349/704] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 4 00:47:23 2022] GET /identity/v3/users/nova => generated 82 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:23.384828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1acf4a5-bb67-4125-82ac-f583cfcb00b8 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:23.393096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:23.429296 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b391774c-ba19-4b63-aeba-381b939fa997 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:23.429535 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b391774c-ba19-4b63-aeba-381b939fa997 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:23.429790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b391774c-ba19-4b63-aeba-381b939fa997 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:23.431344 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 350/705] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:23.442434 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888915ff-05b7-4b45-9c1f-a1cb08fce7fd None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:23.442715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888915ff-05b7-4b45-9c1f-a1cb08fce7fd None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:23.442991 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888915ff-05b7-4b45-9c1f-a1cb08fce7fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:23.467661 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-888915ff-05b7-4b45-9c1f-a1cb08fce7fd None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:23.468862 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-888915ff-05b7-4b45-9c1f-a1cb08fce7fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:23.555376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1acf4a5-bb67-4125-82ac-f583cfcb00b8 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:23.590106 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:23.604257 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 351/706] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:23.611252 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a8a615-d227-42a6-9034-5061fc12a772 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:23.611899 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a8a615-d227-42a6-9034-5061fc12a772 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:23.612409 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a8a615-d227-42a6-9034-5061fc12a772 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:23.612676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a8a615-d227-42a6-9034-5061fc12a772 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:23.613449 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a8a615-d227-42a6-9034-5061fc12a772 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:23.613761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a8a615-d227-42a6-9034-5061fc12a772 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:23.615194 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a8a615-d227-42a6-9034-5061fc12a772 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:23.641337 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 356/707] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 4 00:47:23 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:23.647464 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888915ff-05b7-4b45-9c1f-a1cb08fce7fd None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:23.681967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a8a615-d227-42a6-9034-5061fc12a772 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:23.799794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:23.820659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888915ff-05b7-4b45-9c1f-a1cb08fce7fd None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:23.851500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a8a615-d227-42a6-9034-5061fc12a772 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:23.873210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a136d082-bd52-4f20-b7e4-5045f58d21bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:23.873844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a136d082-bd52-4f20-b7e4-5045f58d21bc None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:23.874127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a136d082-bd52-4f20-b7e4-5045f58d21bc None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:23.874401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a136d082-bd52-4f20-b7e4-5045f58d21bc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:23.896067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1a88f0-6c12-49c5-a2e2-ddbe7199cec2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:23.898279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1a88f0-6c12-49c5-a2e2-ddbe7199cec2 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:23.899006 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1a88f0-6c12-49c5-a2e2-ddbe7199cec2 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:23.900764 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1a88f0-6c12-49c5-a2e2-ddbe7199cec2 None admin] PATH_INFO: `/v3/services/network` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:23.901610 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1a88f0-6c12-49c5-a2e2-ddbe7199cec2 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:23.903287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1a88f0-6c12-49c5-a2e2-ddbe7199cec2 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:23.923831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-bd1a88f0-6c12-49c5-a2e2-ddbe7199cec2 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 04 00:47:23.923831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:23.923831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:23.923831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:23.923831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:23.923831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:23.923831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:23.923831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:23.923831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 04 00:47:23.923831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:23.923831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:23.923831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:23.923831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:23.923831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:23.923831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:23.923831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:23.923831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:23.924923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:23.924923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:23.924923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:23.924923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:23.924923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:23.924923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:23.924923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:23.924923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:23.924923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:23.924923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:23.924923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:23.924923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:23.924923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:23.924923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:23.924923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:23.924923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:23.925797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:23.925797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:23.925797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:23.925797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:23.925797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:23.925797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:23.925797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:23.925797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:23.925797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:23.925797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:23.925797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:23.925797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:23.925797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 04 00:47:23.925797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:23.929397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 357/708] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:23 2022] GET /identity/v3/services/network => generated 88 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:23.930469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1a88f0-6c12-49c5-a2e2-ddbe7199cec2 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:23.976240 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:23.987549 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a136d082-bd52-4f20-b7e4-5045f58d21bc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:23.989366 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a136d082-bd52-4f20-b7e4-5045f58d21bc None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:24.047467 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1a88f0-6c12-49c5-a2e2-ddbe7199cec2 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:24.073846 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:24.082287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05efaa73-bbc7-4fe8-9d0d-35b194bfc89f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:24.083180 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05efaa73-bbc7-4fe8-9d0d-35b194bfc89f None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:24.083695 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05efaa73-bbc7-4fe8-9d0d-35b194bfc89f None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:24.087292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05efaa73-bbc7-4fe8-9d0d-35b194bfc89f None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:24.087292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05efaa73-bbc7-4fe8-9d0d-35b194bfc89f None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:24.087292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05efaa73-bbc7-4fe8-9d0d-35b194bfc89f None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:24.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-05efaa73-bbc7-4fe8-9d0d-35b194bfc89f None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 04 00:47:24.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:24.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:24.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:24.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:24.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:24.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:24.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:24.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 04 00:47:24.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:24.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:24.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:24.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:24.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:24.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:24.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:24.095410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:24.096572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:24.096572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:24.096572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:24.096572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:24.096572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:24.096572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:24.096572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:24.096572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:24.096572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:24.096572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:24.096572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:24.096572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:24.096572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:24.096572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:24.096572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:24.096572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:24.097425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:24.097425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:24.097425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:24.097425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:24.097425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:24.097425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:24.097425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:24.097425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:24.097425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:24.097425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:24.097425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:24.097425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:24.097425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 04 00:47:24.097425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:24.104408 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 352/709] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 454 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:47:24.104408 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a136d082-bd52-4f20-b7e4-5045f58d21bc None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:24.104992 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 358/710] 10.222.0.42 () {56 vars in 1129 bytes} [Sun Sep 4 00:47:23 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:24.106180 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05efaa73-bbc7-4fe8-9d0d-35b194bfc89f None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:24.209264 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:24.217231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a136d082-bd52-4f20-b7e4-5045f58d21bc None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:24.218461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05efaa73-bbc7-4fe8-9d0d-35b194bfc89f None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:24.262137 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30339307-2cde-47e2-8ee6-3ba01d1becc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:24.263113 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30339307-2cde-47e2-8ee6-3ba01d1becc0 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:24.263601 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30339307-2cde-47e2-8ee6-3ba01d1becc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:24.264101 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30339307-2cde-47e2-8ee6-3ba01d1becc0 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:24.264966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30339307-2cde-47e2-8ee6-3ba01d1becc0 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:24.265513 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30339307-2cde-47e2-8ee6-3ba01d1becc0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:24.267152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30339307-2cde-47e2-8ee6-3ba01d1becc0 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:24.278554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 359/711] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Sep 4 00:47:24 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:24.279933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30339307-2cde-47e2-8ee6-3ba01d1becc0 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:24.344667 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:24.378149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef043fd8-e836-49cd-892e-57609a8c3dc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:24.378914 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef043fd8-e836-49cd-892e-57609a8c3dc7 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:24.379197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef043fd8-e836-49cd-892e-57609a8c3dc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:24.379474 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef043fd8-e836-49cd-892e-57609a8c3dc7 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:24.380329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef043fd8-e836-49cd-892e-57609a8c3dc7 None admin] RBAC: Adding query filter params (name=network) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:24.380446 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef043fd8-e836-49cd-892e-57609a8c3dc7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:24.381825 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef043fd8-e836-49cd-892e-57609a8c3dc7 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:24.391615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 353/712] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 4 00:47:24 2022] GET /identity/v3/services?name=network => generated 124 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:24.391615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef043fd8-e836-49cd-892e-57609a8c3dc7 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:24.400155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30339307-2cde-47e2-8ee6-3ba01d1becc0 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:24.441542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d749683-4010-4ebf-ad9d-8154a99bfea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:24.442231 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d749683-4010-4ebf-ad9d-8154a99bfea1 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:24.442511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d749683-4010-4ebf-ad9d-8154a99bfea1 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:24.442782 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d749683-4010-4ebf-ad9d-8154a99bfea1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:24.457589 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d749683-4010-4ebf-ad9d-8154a99bfea1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:24.459374 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d749683-4010-4ebf-ad9d-8154a99bfea1 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:24.480057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-7d749683-4010-4ebf-ad9d-8154a99bfea1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:24.480057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:24.480057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:24.480057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:24.480057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:24.480057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:24.480057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:24.480057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:24.480057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:24.480057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:24.480057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:24.480057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:24.480057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:24.480057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:24.480057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:24.480057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:24.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:24.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:24.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:24.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:24.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:24.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:24.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:24.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:24.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:24.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:24.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:24.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:24.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:24.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:24.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:24.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:24.481009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:24.482575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:24.482575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:24.482575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:24.482575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 360/713] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:24.482575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d749683-4010-4ebf-ad9d-8154a99bfea1 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:24.547664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef043fd8-e836-49cd-892e-57609a8c3dc7 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:24.590500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f76071-340b-4693-95af-7981fe3f5d34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:24.591712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f76071-340b-4693-95af-7981fe3f5d34 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:24.592436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f76071-340b-4693-95af-7981fe3f5d34 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:24.592913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f76071-340b-4693-95af-7981fe3f5d34 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:24.594155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f76071-340b-4693-95af-7981fe3f5d34 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:24.594717 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f76071-340b-4693-95af-7981fe3f5d34 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:24.596860 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f76071-340b-4693-95af-7981fe3f5d34 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:24.605608 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 354/714] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Sep 4 00:47:24 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:24.607079 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f76071-340b-4693-95af-7981fe3f5d34 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:24.616285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d749683-4010-4ebf-ad9d-8154a99bfea1 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:24.628899 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:24.639986 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:24.698595 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb37667-cf84-47ba-a85b-412e81db2bd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:24.699538 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb37667-cf84-47ba-a85b-412e81db2bd4 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:24.699965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb37667-cf84-47ba-a85b-412e81db2bd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:24.700388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb37667-cf84-47ba-a85b-412e81db2bd4 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:24.701249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb37667-cf84-47ba-a85b-412e81db2bd4 None admin] RBAC: Adding query filter params (type=network) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:24.701705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb37667-cf84-47ba-a85b-412e81db2bd4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:24.703298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb37667-cf84-47ba-a85b-412e81db2bd4 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:24.714147 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 361/715] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 4 00:47:24 2022] GET /identity/v3/services?type=network => generated 124 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:24.779669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f76071-340b-4693-95af-7981fe3f5d34 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:24.803705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74483a11-cdb8-49bc-980f-90289470f07c None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:24.803917 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74483a11-cdb8-49bc-980f-90289470f07c None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:24.804197 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74483a11-cdb8-49bc-980f-90289470f07c None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:24.805875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 362/716] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:47:24.811560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b06cb63-9c6c-4203-b4ba-599be9e08d71 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:24.811846 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b06cb63-9c6c-4203-b4ba-599be9e08d71 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:24.812111 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b06cb63-9c6c-4203-b4ba-599be9e08d71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:24.838941 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0982848-e522-4acd-9ef1-57ef62b1d301 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:24.839355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-4b06cb63-9c6c-4203-b4ba-599be9e08d71 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:24.839575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0982848-e522-4acd-9ef1-57ef62b1d301 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:24.839863 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0982848-e522-4acd-9ef1-57ef62b1d301 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:24.840144 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0982848-e522-4acd-9ef1-57ef62b1d301 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:24.840815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0982848-e522-4acd-9ef1-57ef62b1d301 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:24.841152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0982848-e522-4acd-9ef1-57ef62b1d301 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:24.841784 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4b06cb63-9c6c-4203-b4ba-599be9e08d71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:24.842556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0982848-e522-4acd-9ef1-57ef62b1d301 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:24.863605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 355/717] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:24.865377 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e43469a-38f6-47d5-9781-98ecb6231d8a None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:24.865669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e43469a-38f6-47d5-9781-98ecb6231d8a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:24.865948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e43469a-38f6-47d5-9781-98ecb6231d8a None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:24.867230 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 356/718] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:24.873343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd28903a-c19a-4511-9ba4-9db1ae1f78b7 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:24.873734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd28903a-c19a-4511-9ba4-9db1ae1f78b7 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:24.873886 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd28903a-c19a-4511-9ba4-9db1ae1f78b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:24.894246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-cd28903a-c19a-4511-9ba4-9db1ae1f78b7 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:24.895457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-cd28903a-c19a-4511-9ba4-9db1ae1f78b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:24.924875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 363/719] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:24.925931 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b06cb63-9c6c-4203-b4ba-599be9e08d71 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:25.012179 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 357/720] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 04 00:47:25.077129 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:25.080921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b06cb63-9c6c-4203-b4ba-599be9e08d71 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:25.130437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee84545-ca32-41c7-8da6-202944c5c6ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:25.131068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee84545-ca32-41c7-8da6-202944c5c6ba None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:25.131345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee84545-ca32-41c7-8da6-202944c5c6ba None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:25.131618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee84545-ca32-41c7-8da6-202944c5c6ba None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:25.143805 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee84545-ca32-41c7-8da6-202944c5c6ba None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:25.145386 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee84545-ca32-41c7-8da6-202944c5c6ba None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:25.163723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-bee84545-ca32-41c7-8da6-202944c5c6ba None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:25.163723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:25.163723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:25.163723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:25.163723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:25.163723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:25.163723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:25.163723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:25.163723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:25.163723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:25.163723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:25.163723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:25.163723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:25.163723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:25.163723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:25.164679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:25.164679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:25.164679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:25.164679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:25.164679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:25.164679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:25.164679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:25.164679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:25.164679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:25.164679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:25.164679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:25.164679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:25.164679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:25.164679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:25.164679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:25.164679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:25.164679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:25.165630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:25.165630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:25.165630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:25.165630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:25.165630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:25.165630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:25.165630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:25.165630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:25.165630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 364/721] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:25.170729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd28903a-c19a-4511-9ba4-9db1ae1f78b7 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:25.187551 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:25.250181 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee84545-ca32-41c7-8da6-202944c5c6ba None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:25.314289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd28903a-c19a-4511-9ba4-9db1ae1f78b7 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:25.377820 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a973b0d-528f-4582-845f-29bbe6966182 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:25.378546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a973b0d-528f-4582-845f-29bbe6966182 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:25.379022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a973b0d-528f-4582-845f-29bbe6966182 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:25.379298 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a973b0d-528f-4582-845f-29bbe6966182 None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:25.380007 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a973b0d-528f-4582-845f-29bbe6966182 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:25.380312 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a973b0d-528f-4582-845f-29bbe6966182 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:25.381772 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a973b0d-528f-4582-845f-29bbe6966182 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:25.399936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 358/722] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:25.405630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a973b0d-528f-4582-845f-29bbe6966182 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:25.465858 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee84545-ca32-41c7-8da6-202944c5c6ba None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:25.501468 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d467970-970a-491f-8f21-d12c273e57b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:25.502104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d467970-970a-491f-8f21-d12c273e57b5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:25.503031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d467970-970a-491f-8f21-d12c273e57b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:25.503304 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d467970-970a-491f-8f21-d12c273e57b5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:25.515236 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d467970-970a-491f-8f21-d12c273e57b5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:25.521459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-2d467970-970a-491f-8f21-d12c273e57b5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:47:25.521459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:25.521459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:25.521459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:25.521459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:25.521459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:25.521459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:25.521459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:25.521459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:25.521459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:25.521459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:25.521459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:25.521459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:25.521459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:25.521459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:25.521459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:25.522355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:25.522355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:25.522355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:25.522355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:25.522355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:25.522355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:25.522355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:25.522355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:25.522355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:25.522355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:25.522355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:25.522355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:25.522355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:25.522355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:25.522355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:25.522355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:25.523160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:25.523160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:25.523160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:25.523160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:25.523160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:25.523160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:25.523160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:25.523160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:25.523160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:25.523160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:25.523160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:47:25.523160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:25.523780 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 365/723] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:47:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:25.524438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d467970-970a-491f-8f21-d12c273e57b5 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:25.529533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:25.538128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a973b0d-528f-4582-845f-29bbe6966182 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:25.554418 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:25.583461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c409596f-c3ba-4574-92be-f74202f4df46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:25.584215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c409596f-c3ba-4574-92be-f74202f4df46 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:25.584500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c409596f-c3ba-4574-92be-f74202f4df46 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:25.585063 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c409596f-c3ba-4574-92be-f74202f4df46 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:25.585821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c409596f-c3ba-4574-92be-f74202f4df46 None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=57c88f16c7df40adb917f49804fda21f) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:25.586128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c409596f-c3ba-4574-92be-f74202f4df46 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:25.587974 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c409596f-c3ba-4574-92be-f74202f4df46 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:25.597186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 359/724] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:25 2022] GET /identity/v3/role_assignments?user.id=57c88f16c7df40adb917f49804fda21f&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=19c006412bec45dc8febd7cc86b9678f => generated 259 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:25.598101 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c409596f-c3ba-4574-92be-f74202f4df46 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:25.707329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d467970-970a-491f-8f21-d12c273e57b5 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:25.719688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c409596f-c3ba-4574-92be-f74202f4df46 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:25.766480 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2598f9bc-0db3-4bdc-830d-036a424d100c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:25.767500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2598f9bc-0db3-4bdc-830d-036a424d100c None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:25.768610 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2598f9bc-0db3-4bdc-830d-036a424d100c None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:25.769128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2598f9bc-0db3-4bdc-830d-036a424d100c None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:25.770038 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2598f9bc-0db3-4bdc-830d-036a424d100c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:25.770562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2598f9bc-0db3-4bdc-830d-036a424d100c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:25.772177 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2598f9bc-0db3-4bdc-830d-036a424d100c None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:25.779117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:25.781841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6081d5-1432-4ad6-9301-202a01d20779 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:25.782500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6081d5-1432-4ad6-9301-202a01d20779 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:25.782900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6081d5-1432-4ad6-9301-202a01d20779 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:25.783172 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6081d5-1432-4ad6-9301-202a01d20779 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:25.785029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 360/725] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:47:25 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:25.790791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2598f9bc-0db3-4bdc-830d-036a424d100c None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:25.804774 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6081d5-1432-4ad6-9301-202a01d20779 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:25.810737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-bc6081d5-1432-4ad6-9301-202a01d20779 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:25.810737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:25.810737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:25.810737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:25.810737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:25.810737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:25.810737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:25.810737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:25.810737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:25.810737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:25.810737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:25.810737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:25.810737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:25.810737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:25.810737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:25.810737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:25.811681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:25.811681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:25.811681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:25.811681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:25.811681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:25.811681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:25.811681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:25.811681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:25.811681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:25.811681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:25.811681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:25.811681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:25.811681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:25.811681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:25.811681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:25.811681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:25.812494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:25.812494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:25.812494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:25.812494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:25.812494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:25.812494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:25.812494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:25.812494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:25.812494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:25.812494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:25.812494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:25.812494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:25.812494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 366/726] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:25 2022] GET /identity/v3/roles/service => generated 85 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:25.817853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6081d5-1432-4ad6-9301-202a01d20779 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:25.923315 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2598f9bc-0db3-4bdc-830d-036a424d100c None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:25.960461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc2ef5f-1529-4648-b27e-b5e1f5d1e619 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:25.966780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc2ef5f-1529-4648-b27e-b5e1f5d1e619 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:25.966780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc2ef5f-1529-4648-b27e-b5e1f5d1e619 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:25.966780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc2ef5f-1529-4648-b27e-b5e1f5d1e619 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:25.977454 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:25.982970 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc2ef5f-1529-4648-b27e-b5e1f5d1e619 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:25.985797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc2ef5f-1529-4648-b27e-b5e1f5d1e619 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:26.010127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-7bc2ef5f-1529-4648-b27e-b5e1f5d1e619 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:26.010127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:26.010127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:26.010127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:26.010127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:26.010127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:26.010127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:26.010127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:26.010127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:26.010127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:26.010127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:26.010127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:26.010127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:26.010127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:26.010127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:26.010127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:26.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:26.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:26.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:26.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:26.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:26.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:26.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:26.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:26.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:26.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:26.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:26.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:26.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:26.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:26.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:26.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:26.011103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:26.012039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:26.012039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:26.012039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:26.012039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 361/727] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:26.019313 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc2ef5f-1529-4648-b27e-b5e1f5d1e619 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:26.034355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6081d5-1432-4ad6-9301-202a01d20779 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:26.098203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18060ed-ca5f-4be4-9f95-867b4c656746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:26.099519 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18060ed-ca5f-4be4-9f95-867b4c656746 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:26.100756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18060ed-ca5f-4be4-9f95-867b4c656746 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:26.101258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18060ed-ca5f-4be4-9f95-867b4c656746 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:26.102861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18060ed-ca5f-4be4-9f95-867b4c656746 None admin] RBAC: Adding query filter params (name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:26.103551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18060ed-ca5f-4be4-9f95-867b4c656746 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:26.106309 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18060ed-ca5f-4be4-9f95-867b4c656746 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:26.117432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:26.121935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 367/728] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:26.125990 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18060ed-ca5f-4be4-9f95-867b4c656746 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:26.201316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc2ef5f-1529-4648-b27e-b5e1f5d1e619 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:26.263038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18060ed-ca5f-4be4-9f95-867b4c656746 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:26.271179 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5bcae4-ca82-4c7d-8195-20e1fefd43d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:26.272138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5bcae4-ca82-4c7d-8195-20e1fefd43d9 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:26.272271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5bcae4-ca82-4c7d-8195-20e1fefd43d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:26.272486 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5bcae4-ca82-4c7d-8195-20e1fefd43d9 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:26.273179 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5bcae4-ca82-4c7d-8195-20e1fefd43d9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:26.273491 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5bcae4-ca82-4c7d-8195-20e1fefd43d9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:26.275079 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5bcae4-ca82-4c7d-8195-20e1fefd43d9 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:26.298467 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 362/729] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:26.305972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc432677-9c1e-477d-97a2-4eb80d69a2f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:26.306890 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc432677-9c1e-477d-97a2-4eb80d69a2f6 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:26.307371 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc432677-9c1e-477d-97a2-4eb80d69a2f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:26.307907 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc432677-9c1e-477d-97a2-4eb80d69a2f6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:26.310257 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5bcae4-ca82-4c7d-8195-20e1fefd43d9 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:26.321059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc432677-9c1e-477d-97a2-4eb80d69a2f6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:26.322913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc432677-9c1e-477d-97a2-4eb80d69a2f6 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:26.325364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:26.339534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-fc432677-9c1e-477d-97a2-4eb80d69a2f6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:26.339534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:26.339534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:26.339534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:26.339534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:26.339534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:26.339534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:26.339534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:26.339534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:26.339534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:26.339534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:26.339534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:26.339534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:26.339534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:26.339534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:26.339534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:26.351769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:26.351769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:26.351769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:26.351769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:26.351769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:26.351769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:26.351769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:26.351769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:26.351769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:26.351769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:26.351769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:26.351769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:26.351769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:26.351769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:26.351769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:26.351769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:26.351769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:26.352679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:26.352679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:26.352679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:26.352679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 368/730] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:26.352679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc432677-9c1e-477d-97a2-4eb80d69a2f6 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:26.464557 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc432677-9c1e-477d-97a2-4eb80d69a2f6 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:26.500542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee14aef4-3586-423c-af54-6900cef4ed0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:26.501216 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee14aef4-3586-423c-af54-6900cef4ed0d None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:26.501597 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee14aef4-3586-423c-af54-6900cef4ed0d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:26.501883 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee14aef4-3586-423c-af54-6900cef4ed0d None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:26.503267 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee14aef4-3586-423c-af54-6900cef4ed0d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:26.503582 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee14aef4-3586-423c-af54-6900cef4ed0d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:26.504962 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee14aef4-3586-423c-af54-6900cef4ed0d None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:26.521486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 369/731] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:26.523489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee14aef4-3586-423c-af54-6900cef4ed0d None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:26.546949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5bcae4-ca82-4c7d-8195-20e1fefd43d9 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:26.573483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:26.603778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ccfc37-d8a9-49d4-bcd2-59a65eeb3133 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:26.604475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ccfc37-d8a9-49d4-bcd2-59a65eeb3133 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:26.604763 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ccfc37-d8a9-49d4-bcd2-59a65eeb3133 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:26.605036 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ccfc37-d8a9-49d4-bcd2-59a65eeb3133 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:26.613359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ccfc37-d8a9-49d4-bcd2-59a65eeb3133 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:26.614813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ccfc37-d8a9-49d4-bcd2-59a65eeb3133 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:26.629225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-c3ccfc37-d8a9-49d4-bcd2-59a65eeb3133 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 04 00:47:26.629225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:26.629225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:26.629225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:26.629225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:26.629225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:26.629225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:26.629225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:26.629225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:26.629225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:26.629225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:26.629225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:26.629225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:26.629225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:26.629225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:26.629225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:26.630210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:26.631134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:26.631134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:26.631134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:26.631134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:26.631134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:26.631134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:26.631134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:26.631134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:26.631134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 04 00:47:26.631134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:26.631134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 363/732] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:47:26 2022] GET /identity/v3/users/swift => generated 83 bytes in 321 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:26.640243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ccfc37-d8a9-49d4-bcd2-59a65eeb3133 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:26.648022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:26.693526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee14aef4-3586-423c-af54-6900cef4ed0d None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:26.754391 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d2c9e8-2c90-4bc3-a147-51410b97a23f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:26.755037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d2c9e8-2c90-4bc3-a147-51410b97a23f None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:26.755318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d2c9e8-2c90-4bc3-a147-51410b97a23f None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:26.755596 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d2c9e8-2c90-4bc3-a147-51410b97a23f None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:26.769126 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d2c9e8-2c90-4bc3-a147-51410b97a23f None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:26.773455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ccfc37-d8a9-49d4-bcd2-59a65eeb3133 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:26.778692 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d2c9e8-2c90-4bc3-a147-51410b97a23f None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:26.799673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-a3d2c9e8-2c90-4bc3-a147-51410b97a23f None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 04 00:47:26.799673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:26.799673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:26.799673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:26.799673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:26.799673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:26.799673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:26.799673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:26.799673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:26.799673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:26.799673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:26.799673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:26.799673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:26.799673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:26.799673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:26.799673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:26.800742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:26.801699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:26.801699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:26.801699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:26.801699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:26.801699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:26.801699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:26.801699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:26.801699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:26.801699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 04 00:47:26.801699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:26.802216 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 370/733] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 4 00:47:26 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:26.805112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715c6d53-d39e-47db-86f7-c0139f35f294 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:26.805112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715c6d53-d39e-47db-86f7-c0139f35f294 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:26.805112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715c6d53-d39e-47db-86f7-c0139f35f294 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:26.805112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 371/734] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:26 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:26.810751 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4814ef-fe22-42f1-9711-8557546dec48 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:26.811136 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4814ef-fe22-42f1-9711-8557546dec48 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:26.811414 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4814ef-fe22-42f1-9711-8557546dec48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:26.816282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0364fde4-ee2d-41a1-9146-9a4cec71d02a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:26.816889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0364fde4-ee2d-41a1-9146-9a4cec71d02a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:26.817206 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0364fde4-ee2d-41a1-9146-9a4cec71d02a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:26.817481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0364fde4-ee2d-41a1-9146-9a4cec71d02a None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:26.818226 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0364fde4-ee2d-41a1-9146-9a4cec71d02a None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:26.818536 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0364fde4-ee2d-41a1-9146-9a4cec71d02a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:26.819969 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0364fde4-ee2d-41a1-9146-9a4cec71d02a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:26.830443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-2e4814ef-fe22-42f1-9711-8557546dec48 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:26.832144 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-2e4814ef-fe22-42f1-9711-8557546dec48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:26.838638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 364/735] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Sep 4 00:47:26 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:26.839571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0364fde4-ee2d-41a1-9146-9a4cec71d02a None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:26.874258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:26.929203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 372/736] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:26.930584 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4814ef-fe22-42f1-9711-8557546dec48 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:27.002966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0364fde4-ee2d-41a1-9146-9a4cec71d02a None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:27.029068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:27.090879 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef31a26-143b-4451-bf05-9bf81665cac0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:27.091591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef31a26-143b-4451-bf05-9bf81665cac0 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:27.091786 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef31a26-143b-4451-bf05-9bf81665cac0 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:27.092064 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef31a26-143b-4451-bf05-9bf81665cac0 None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:27.092807 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef31a26-143b-4451-bf05-9bf81665cac0 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:27.093138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef31a26-143b-4451-bf05-9bf81665cac0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:27.093274 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4814ef-fe22-42f1-9711-8557546dec48 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:27.094887 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef31a26-143b-4451-bf05-9bf81665cac0 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:27.131498 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 365/737] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Sep 4 00:47:26 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:27.131685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21627b7-c217-4e01-8275-35f9d54f09d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:27.132179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21627b7-c217-4e01-8275-35f9d54f09d8 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:27.132565 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21627b7-c217-4e01-8275-35f9d54f09d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:27.132849 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21627b7-c217-4e01-8275-35f9d54f09d8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:27.137050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef31a26-143b-4451-bf05-9bf81665cac0 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:27.215049 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21627b7-c217-4e01-8275-35f9d54f09d8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:27.217201 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21627b7-c217-4e01-8275-35f9d54f09d8 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:27.258391 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:27.268959 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:27.273343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef31a26-143b-4451-bf05-9bf81665cac0 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:27.309350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cc8095-ea35-4022-9ac6-b6aacf49feb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:27.309961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cc8095-ea35-4022-9ac6-b6aacf49feb9 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:27.310726 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cc8095-ea35-4022-9ac6-b6aacf49feb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:27.310726 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cc8095-ea35-4022-9ac6-b6aacf49feb9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:27.325554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 373/738] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:47:27.327158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de56d882-8897-47ac-adb1-4fb6de4230f1 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:27.327653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de56d882-8897-47ac-adb1-4fb6de4230f1 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:27.328231 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de56d882-8897-47ac-adb1-4fb6de4230f1 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:27.334064 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 374/739] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:27 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:27.335497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de56d882-8897-47ac-adb1-4fb6de4230f1 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:27.388236 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cc8095-ea35-4022-9ac6-b6aacf49feb9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:27.389778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cc8095-ea35-4022-9ac6-b6aacf49feb9 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:27.450605 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de56d882-8897-47ac-adb1-4fb6de4230f1 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:27.482180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:27.492516 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de0aa64-d18a-4296-9ce1-4b5b08f2a85e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:27.493973 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de0aa64-d18a-4296-9ce1-4b5b08f2a85e None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:27.493973 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de0aa64-d18a-4296-9ce1-4b5b08f2a85e None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:27.493973 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de0aa64-d18a-4296-9ce1-4b5b08f2a85e None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:27.494393 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de0aa64-d18a-4296-9ce1-4b5b08f2a85e None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:27.495692 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de0aa64-d18a-4296-9ce1-4b5b08f2a85e None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:27.504156 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 366/740] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 04 00:47:27.504156 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cc8095-ea35-4022-9ac6-b6aacf49feb9 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:27.599145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 375/741] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 4 00:47:27 2022] POST /identity/v3/services => generated 251 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:47:27.599656 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cc8095-ea35-4022-9ac6-b6aacf49feb9 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:27.601247 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2dc64e-b7aa-4c31-822b-582a993f16e4 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:27.601733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2dc64e-b7aa-4c31-822b-582a993f16e4 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:27.602170 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2dc64e-b7aa-4c31-822b-582a993f16e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:27.610099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:27.624179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-af2dc64e-b7aa-4c31-822b-582a993f16e4 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:27.626168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-af2dc64e-b7aa-4c31-822b-582a993f16e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:27.666404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e908345-fd3f-4f39-9a0f-fd7633411953 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:27.668029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e908345-fd3f-4f39-9a0f-fd7633411953 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:27.668611 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e908345-fd3f-4f39-9a0f-fd7633411953 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:27.669191 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e908345-fd3f-4f39-9a0f-fd7633411953 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:27.684665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e908345-fd3f-4f39-9a0f-fd7633411953 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:27.687159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e908345-fd3f-4f39-9a0f-fd7633411953 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:27.708987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-8e908345-fd3f-4f39-9a0f-fd7633411953 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:27.708987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:27.708987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:27.708987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:27.708987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:27.708987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:27.708987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:27.708987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:27.708987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:27.708987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:27.708987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:27.708987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:27.708987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:27.708987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:27.708987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:27.708987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:27.710199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:27.710199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:27.710199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:27.710199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:27.710199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:27.710199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:27.710199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:27.710199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:27.710199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:27.710199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:27.710199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:27.710199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:27.710199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:27.710199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:27.710199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:27.710199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:27.710199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:27.711262 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:27.711262 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:27.711262 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:27.713047 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 367/742] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:27.714204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e908345-fd3f-4f39-9a0f-fd7633411953 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:27.717240 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 376/743] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:27 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:27.720956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2dc64e-b7aa-4c31-822b-582a993f16e4 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:27.780609 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:27.850699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2dc64e-b7aa-4c31-822b-582a993f16e4 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:27.852982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e908345-fd3f-4f39-9a0f-fd7633411953 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:27.888358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3fd41d-cd5a-4aa3-be84-bcbba49b6e3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:27.888897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3fd41d-cd5a-4aa3-be84-bcbba49b6e3e None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:27.889232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3fd41d-cd5a-4aa3-be84-bcbba49b6e3e None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:27.889510 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3fd41d-cd5a-4aa3-be84-bcbba49b6e3e None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:27.890172 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3fd41d-cd5a-4aa3-be84-bcbba49b6e3e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:27.890481 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3fd41d-cd5a-4aa3-be84-bcbba49b6e3e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:27.891652 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55adcfb3-c788-4c14-bb24-641e24e840dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:27.891850 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3fd41d-cd5a-4aa3-be84-bcbba49b6e3e None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:27.892290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55adcfb3-c788-4c14-bb24-641e24e840dd None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:27.892573 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55adcfb3-c788-4c14-bb24-641e24e840dd None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:27.892890 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55adcfb3-c788-4c14-bb24-641e24e840dd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:27.893661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:27.904375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55adcfb3-c788-4c14-bb24-641e24e840dd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:27.906089 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55adcfb3-c788-4c14-bb24-641e24e840dd None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:27.906211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 377/744] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:27.907699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41948f0a-6b03-4c8e-bf3c-d702ffcc5458 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:27.908044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41948f0a-6b03-4c8e-bf3c-d702ffcc5458 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:27.908572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41948f0a-6b03-4c8e-bf3c-d702ffcc5458 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:27.910523 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 378/745] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:27.914808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41948f0a-6b03-4c8e-bf3c-d702ffcc5458 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:27.923600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-55adcfb3-c788-4c14-bb24-641e24e840dd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:27.923600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:27.923600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:27.923600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:27.923600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:27.923600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:27.923600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:27.923600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:27.923600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:27.923600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:27.923600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:27.923600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:27.923600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:27.923600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:27.923600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:27.923600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:27.925463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:27.925463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:27.925463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:27.925463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:27.925463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:27.925463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:27.925463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:27.925463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:27.925463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:27.925463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:27.925463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:27.925463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:27.925463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:27.925463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:27.925463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:27.925463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:27.925463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:27.926520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:27.926520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:27.926520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:27.928588 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 368/746] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:27.930044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb928b26-9af5-40e5-8933-bb38fc53319d None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:27.930539 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb928b26-9af5-40e5-8933-bb38fc53319d None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:27.931029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb928b26-9af5-40e5-8933-bb38fc53319d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:27.952131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-fb928b26-9af5-40e5-8933-bb38fc53319d None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:27.953387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-fb928b26-9af5-40e5-8933-bb38fc53319d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:28.037765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 369/747] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:27 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 04 00:47:28.038713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb928b26-9af5-40e5-8933-bb38fc53319d None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:28.039259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41948f0a-6b03-4c8e-bf3c-d702ffcc5458 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:28.060456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:28.077755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474ed424-d565-4cda-92bb-4761b82956fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:28.078551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474ed424-d565-4cda-92bb-4761b82956fc None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:28.078930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474ed424-d565-4cda-92bb-4761b82956fc None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:28.079303 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474ed424-d565-4cda-92bb-4761b82956fc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:28.087355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474ed424-d565-4cda-92bb-4761b82956fc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:28.090067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474ed424-d565-4cda-92bb-4761b82956fc None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:28.104036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-474ed424-d565-4cda-92bb-4761b82956fc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:28.104036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:28.104036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:28.104036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:28.104036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:28.104036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:28.104036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:28.104036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:28.104036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:28.104036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:28.104036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:28.104036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:28.104036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:28.104036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:28.104036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:28.105135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:28.105135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:28.105135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:28.105135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:28.105135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:28.105135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:28.105135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:28.105135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:28.105135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:28.105135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:28.105135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:28.105135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:28.105135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:28.105135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:28.105135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:28.105135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:28.105135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:28.106847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:28.106847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:28.106847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:28.106847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:28.106847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:28.106847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:28.106847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:28.106847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:28.112370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 379/748] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:28.113318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474ed424-d565-4cda-92bb-4761b82956fc None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:28.159059 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb928b26-9af5-40e5-8933-bb38fc53319d None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:28.174673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:28.202721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fc606d-ff96-4a08-8ccf-0742bbeb8bd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:28.203487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fc606d-ff96-4a08-8ccf-0742bbeb8bd1 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:28.204138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fc606d-ff96-4a08-8ccf-0742bbeb8bd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:28.204430 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fc606d-ff96-4a08-8ccf-0742bbeb8bd1 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:28.205443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fc606d-ff96-4a08-8ccf-0742bbeb8bd1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:28.205888 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fc606d-ff96-4a08-8ccf-0742bbeb8bd1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:28.208090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fc606d-ff96-4a08-8ccf-0742bbeb8bd1 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:28.226470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 370/749] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:28.227330 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fc606d-ff96-4a08-8ccf-0742bbeb8bd1 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:28.229364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474ed424-d565-4cda-92bb-4761b82956fc None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:28.270811 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0cd5f1-4a38-4a6e-9be3-14f7fd9d5738 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:28.271516 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0cd5f1-4a38-4a6e-9be3-14f7fd9d5738 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:28.271856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0cd5f1-4a38-4a6e-9be3-14f7fd9d5738 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:28.272241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0cd5f1-4a38-4a6e-9be3-14f7fd9d5738 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:28.273058 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0cd5f1-4a38-4a6e-9be3-14f7fd9d5738 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:28.274698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0cd5f1-4a38-4a6e-9be3-14f7fd9d5738 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:28.322325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:28.345481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fc606d-ff96-4a08-8ccf-0742bbeb8bd1 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:28.353278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 380/750] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 4 00:47:28 2022] POST /identity/v3/services => generated 240 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:47:28.353278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0cd5f1-4a38-4a6e-9be3-14f7fd9d5738 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:28.395432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52bdd73-c1c4-4a87-8654-b69f3cd455a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:28.395953 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52bdd73-c1c4-4a87-8654-b69f3cd455a2 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:28.396302 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52bdd73-c1c4-4a87-8654-b69f3cd455a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:28.396581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52bdd73-c1c4-4a87-8654-b69f3cd455a2 None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:28.397371 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52bdd73-c1c4-4a87-8654-b69f3cd455a2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:28.397694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52bdd73-c1c4-4a87-8654-b69f3cd455a2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:28.399211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52bdd73-c1c4-4a87-8654-b69f3cd455a2 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:28.418056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 371/751] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:28.418056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52bdd73-c1c4-4a87-8654-b69f3cd455a2 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:28.523248 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0cd5f1-4a38-4a6e-9be3-14f7fd9d5738 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:28.543105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:28.549648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52bdd73-c1c4-4a87-8654-b69f3cd455a2 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:28.569458 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a891c6-1589-44f1-b749-521f4c80019b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:28.570483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a891c6-1589-44f1-b749-521f4c80019b None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:28.570483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a891c6-1589-44f1-b749-521f4c80019b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:28.570672 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a891c6-1589-44f1-b749-521f4c80019b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:28.579364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a891c6-1589-44f1-b749-521f4c80019b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:28.581727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a891c6-1589-44f1-b749-521f4c80019b None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:28.588234 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:28.593355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-85a891c6-1589-44f1-b749-521f4c80019b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:28.593355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:28.593355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:28.593355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:28.593355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:28.593355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:28.593355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:28.593355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:28.593355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:28.593355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:28.593355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:28.593355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:28.593355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:28.593355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:28.593355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:28.594226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:28.594226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:28.594226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:28.594226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:28.594226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:28.594226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:28.594226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:28.594226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:28.594226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:28.594226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:28.594226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:28.594226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:28.594226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:28.594226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:28.594226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:28.594226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:28.594226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:28.595087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:28.595087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:28.595087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:28.595087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:28.595087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:28.595087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:28.595087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:28.595087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:28.597306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dfaf2f-fc0f-471f-9357-a31e4853b8b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:28.597946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 381/752] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:28.599102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dfaf2f-fc0f-471f-9357-a31e4853b8b5 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:28.599102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dfaf2f-fc0f-471f-9357-a31e4853b8b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:28.599283 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a891c6-1589-44f1-b749-521f4c80019b None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:28.599690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dfaf2f-fc0f-471f-9357-a31e4853b8b5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:28.614378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53dfaf2f-fc0f-471f-9357-a31e4853b8b5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:28.621060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-53dfaf2f-fc0f-471f-9357-a31e4853b8b5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:47:28.621060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:28.621060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:28.621060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:28.621060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:28.621060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:28.621060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:28.621060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:28.621060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:28.621060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:28.621060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:28.621060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:28.621060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:28.621060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:28.621060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:28.621060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:28.621982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:28.621982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:28.621982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:28.621982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:28.621982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:28.621982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:28.621982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:28.621982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:28.621982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:28.621982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:28.621982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:28.621982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:28.621982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:28.621982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:28.621982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:28.621982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:28.622814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:28.622814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:28.622814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:28.622814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:28.622814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:28.622814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:28.622814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:28.622814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:28.622814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:28.622814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:28.622814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:47:28.622814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:28.622814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 372/753] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:47:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:28.623764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dfaf2f-fc0f-471f-9357-a31e4853b8b5 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:28.737925 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a891c6-1589-44f1-b749-521f4c80019b None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:28.753949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dfaf2f-fc0f-471f-9357-a31e4853b8b5 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:28.780499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb011680-20d0-4ca6-af81-323e65c8490c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:28.781203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb011680-20d0-4ca6-af81-323e65c8490c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:28.781487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb011680-20d0-4ca6-af81-323e65c8490c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:28.781761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb011680-20d0-4ca6-af81-323e65c8490c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:28.782490 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb011680-20d0-4ca6-af81-323e65c8490c None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=123ab6ff796c4e599ce1d6f4095bd898) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:28.782798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb011680-20d0-4ca6-af81-323e65c8490c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:28.784797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb011680-20d0-4ca6-af81-323e65c8490c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:28.790894 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 382/754] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:28 2022] GET /identity/v3/role_assignments?user.id=123ab6ff796c4e599ce1d6f4095bd898&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=19c006412bec45dc8febd7cc86b9678f => generated 259 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:28.791783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb011680-20d0-4ca6-af81-323e65c8490c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:28.798919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7cacda-42d6-4167-9571-53e77f8101cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:28.799596 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7cacda-42d6-4167-9571-53e77f8101cb None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:28.799873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7cacda-42d6-4167-9571-53e77f8101cb None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:28.800146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7cacda-42d6-4167-9571-53e77f8101cb None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:28.800874 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7cacda-42d6-4167-9571-53e77f8101cb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:28.802289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7cacda-42d6-4167-9571-53e77f8101cb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:28.809136 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7cacda-42d6-4167-9571-53e77f8101cb None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:28.831325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 373/755] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:28.842504 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7cacda-42d6-4167-9571-53e77f8101cb None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:28.861783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:28.938038 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:28.975354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb011680-20d0-4ca6-af81-323e65c8490c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:29.009612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7cacda-42d6-4167-9571-53e77f8101cb None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:29.042410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc80b0ff-4950-40f6-aa33-64393695ea4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:29.043032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc80b0ff-4950-40f6-aa33-64393695ea4a None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:29.043313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc80b0ff-4950-40f6-aa33-64393695ea4a None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:29.043590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc80b0ff-4950-40f6-aa33-64393695ea4a None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:29.044266 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc80b0ff-4950-40f6-aa33-64393695ea4a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:29.044579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc80b0ff-4950-40f6-aa33-64393695ea4a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:29.050044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc80b0ff-4950-40f6-aa33-64393695ea4a None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:29.056205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6908165c-b456-4093-a7b6-60af5d7282df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:29.057055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6908165c-b456-4093-a7b6-60af5d7282df None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:29.057550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6908165c-b456-4093-a7b6-60af5d7282df None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:29.058050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6908165c-b456-4093-a7b6-60af5d7282df None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:29.058964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6908165c-b456-4093-a7b6-60af5d7282df None admin] RBAC: Adding query filter params (role.id=df912a77fbaa479db172f76a214f8bc8, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=70cef7c271e046a3b49230057cc2d28f) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:29.059461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6908165c-b456-4093-a7b6-60af5d7282df None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:29.061303 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6908165c-b456-4093-a7b6-60af5d7282df None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:29.068525 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 374/756] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:28 2022] GET /identity/v3/role_assignments?user.id=70cef7c271e046a3b49230057cc2d28f&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=df912a77fbaa479db172f76a214f8bc8 => generated 259 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:29.074596 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 383/757] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:47:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:29.080476 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6908165c-b456-4093-a7b6-60af5d7282df None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:29.232449 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6908165c-b456-4093-a7b6-60af5d7282df None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:29.255829 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:29.277871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4a28fa-a3c8-4016-b3d1-fda60c8dd259 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:29.278784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4a28fa-a3c8-4016-b3d1-fda60c8dd259 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:29.279297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4a28fa-a3c8-4016-b3d1-fda60c8dd259 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:29.279767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4a28fa-a3c8-4016-b3d1-fda60c8dd259 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:29.293483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4a28fa-a3c8-4016-b3d1-fda60c8dd259 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:29.295152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4a28fa-a3c8-4016-b3d1-fda60c8dd259 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:29.310926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-0b4a28fa-a3c8-4016-b3d1-fda60c8dd259 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:29.310926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:29.310926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:29.310926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:29.310926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:29.310926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:29.310926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:29.310926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:29.310926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:29.310926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:29.310926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:29.310926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:29.310926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:29.310926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:29.310926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:29.310926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:29.311906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:29.311906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:29.311906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:29.311906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:29.311906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:29.311906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:29.311906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:29.311906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:29.311906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:29.311906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:29.311906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:29.311906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:29.311906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:29.311906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:29.311906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:29.311906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:29.311906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:29.312781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:29.312781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:29.312781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:29.315236 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 375/758] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:29.321924 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc80b0ff-4950-40f6-aa33-64393695ea4a None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:29.480885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:29.483190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc80b0ff-4950-40f6-aa33-64393695ea4a None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:29.534852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be7a6bc-6f95-4bbf-8d35-49e3dc2fcd27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:29.535391 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be7a6bc-6f95-4bbf-8d35-49e3dc2fcd27 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:29.535667 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be7a6bc-6f95-4bbf-8d35-49e3dc2fcd27 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:29.535936 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be7a6bc-6f95-4bbf-8d35-49e3dc2fcd27 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:29.536588 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be7a6bc-6f95-4bbf-8d35-49e3dc2fcd27 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:29.536886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be7a6bc-6f95-4bbf-8d35-49e3dc2fcd27 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:29.538280 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be7a6bc-6f95-4bbf-8d35-49e3dc2fcd27 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:29.555653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 384/759] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:29.561404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4a28fa-a3c8-4016-b3d1-fda60c8dd259 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:29.713030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4a28fa-a3c8-4016-b3d1-fda60c8dd259 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:29.766283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05e96d2-f5e7-43b6-8502-bc88bb339e82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:29.766920 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05e96d2-f5e7-43b6-8502-bc88bb339e82 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:29.767200 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05e96d2-f5e7-43b6-8502-bc88bb339e82 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:29.767474 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05e96d2-f5e7-43b6-8502-bc88bb339e82 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:29.782037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05e96d2-f5e7-43b6-8502-bc88bb339e82 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:29.783529 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05e96d2-f5e7-43b6-8502-bc88bb339e82 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:29.798432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-c05e96d2-f5e7-43b6-8502-bc88bb339e82 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:47:29.798432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:29.798432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:29.798432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:29.798432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:29.798432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:29.798432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:29.798432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:29.798432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:29.798432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:29.798432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:29.798432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:29.798432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:29.798432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:29.798432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:29.798432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:29.799595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:29.800723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:29.800723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:29.800723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:29.800723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:29.800723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:29.800723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:29.800723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:29.800723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:29.800723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:47:29.800723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:29.800723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 376/760] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 4 00:47:29 2022] GET /identity/v3/users/nova => generated 82 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:29.809607 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be7a6bc-6f95-4bbf-8d35-49e3dc2fcd27 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:29.969290 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be7a6bc-6f95-4bbf-8d35-49e3dc2fcd27 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:30.021794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfb3dce-265c-474f-a471-f5fe78163476 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:30.022416 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb3dce-265c-474f-a471-f5fe78163476 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:30.022702 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb3dce-265c-474f-a471-f5fe78163476 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:30.022981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb3dce-265c-474f-a471-f5fe78163476 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:30.023724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfb3dce-265c-474f-a471-f5fe78163476 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:30.024032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfb3dce-265c-474f-a471-f5fe78163476 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:30.025507 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfb3dce-265c-474f-a471-f5fe78163476 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:30.034493 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae81695f-8707-4020-bd2d-7efe2d7061ca None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:30.034867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae81695f-8707-4020-bd2d-7efe2d7061ca None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:30.035017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae81695f-8707-4020-bd2d-7efe2d7061ca None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:30.036576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 377/761] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:30.041922 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717c4417-4db5-4fc1-af88-03dcb02edc18 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:30.042268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717c4417-4db5-4fc1-af88-03dcb02edc18 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:30.042555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717c4417-4db5-4fc1-af88-03dcb02edc18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:30.047619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 385/762] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 4 00:47:29 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:30.052980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfb3dce-265c-474f-a471-f5fe78163476 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:30.062882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-717c4417-4db5-4fc1-af88-03dcb02edc18 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:30.064226 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-717c4417-4db5-4fc1-af88-03dcb02edc18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:30.080956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:30.097126 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:30.179351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 378/763] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:30 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:30.213032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfb3dce-265c-474f-a471-f5fe78163476 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:30.251769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2ddfca-bf6a-4951-b0f2-c57772a25e43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:30.252555 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2ddfca-bf6a-4951-b0f2-c57772a25e43 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:30.252835 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2ddfca-bf6a-4951-b0f2-c57772a25e43 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:30.253127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2ddfca-bf6a-4951-b0f2-c57772a25e43 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:30.264291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2ddfca-bf6a-4951-b0f2-c57772a25e43 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:30.265783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2ddfca-bf6a-4951-b0f2-c57772a25e43 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:30.281967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-0a2ddfca-bf6a-4951-b0f2-c57772a25e43 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:30.281967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:30.281967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:30.281967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:30.281967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:30.281967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:30.281967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:30.281967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:30.281967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:30.281967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:30.281967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:30.281967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:30.281967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:30.281967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:30.281967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:30.281967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:30.282947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:30.282947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:30.282947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:30.282947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:30.282947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:30.282947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:30.282947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:30.282947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:30.282947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:30.282947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:30.282947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:30.282947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:30.282947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:30.282947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:30.282947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:30.282947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:30.282947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:30.284848 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:30.284848 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:30.284848 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:30.288533 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 386/764] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:30.289925 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717c4417-4db5-4fc1-af88-03dcb02edc18 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:30.423025 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717c4417-4db5-4fc1-af88-03dcb02edc18 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:30.468890 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7491981-bbc2-4be5-9259-9c574abc27dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:30.469543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7491981-bbc2-4be5-9259-9c574abc27dd None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:30.469962 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7491981-bbc2-4be5-9259-9c574abc27dd None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:30.470235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7491981-bbc2-4be5-9259-9c574abc27dd None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:30.470985 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7491981-bbc2-4be5-9259-9c574abc27dd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:30.471356 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7491981-bbc2-4be5-9259-9c574abc27dd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:30.472713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7491981-bbc2-4be5-9259-9c574abc27dd None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:30.489188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 379/765] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:30.490706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2ddfca-bf6a-4951-b0f2-c57772a25e43 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:30.539913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7491981-bbc2-4be5-9259-9c574abc27dd None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:30.559888 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:30.567517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:30.602387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2ddfca-bf6a-4951-b0f2-c57772a25e43 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:30.637215 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84e82a1-c5a4-42ad-9bbd-3e6de069d699 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:30.638039 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84e82a1-c5a4-42ad-9bbd-3e6de069d699 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:30.638418 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84e82a1-c5a4-42ad-9bbd-3e6de069d699 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:30.638797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84e82a1-c5a4-42ad-9bbd-3e6de069d699 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:30.646632 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84e82a1-c5a4-42ad-9bbd-3e6de069d699 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:30.648207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84e82a1-c5a4-42ad-9bbd-3e6de069d699 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:30.660314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-b84e82a1-c5a4-42ad-9bbd-3e6de069d699 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:30.660314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:30.660314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:30.660314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:30.660314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:30.660314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:30.660314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:30.660314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:30.660314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:30.660314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:30.660314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:30.660314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:30.660314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:30.660314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:30.660314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:30.661450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:30.661450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:30.661450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:30.661450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:30.661450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:30.661450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:30.661450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:30.661450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:30.661450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:30.661450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:30.661450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:30.661450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:30.661450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:30.661450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:30.661450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:30.661450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:30.661450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:30.662551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:30.662551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:30.662551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:30.662551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:30.662551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:30.662551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:30.662551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:30.662551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:30.668634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 387/766] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:30.670035 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84e82a1-c5a4-42ad-9bbd-3e6de069d699 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:30.775104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7491981-bbc2-4be5-9259-9c574abc27dd None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:30.776341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84e82a1-c5a4-42ad-9bbd-3e6de069d699 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:30.811486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2010f9fc-12be-4efa-8a74-ff0f50c1282b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:30.812259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2010f9fc-12be-4efa-8a74-ff0f50c1282b None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:30.812648 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2010f9fc-12be-4efa-8a74-ff0f50c1282b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:30.813023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2010f9fc-12be-4efa-8a74-ff0f50c1282b None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:30.817098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2010f9fc-12be-4efa-8a74-ff0f50c1282b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:30.817098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2010f9fc-12be-4efa-8a74-ff0f50c1282b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:30.817098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2010f9fc-12be-4efa-8a74-ff0f50c1282b None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:30.833480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:30.833797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 388/767] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:30.836166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2010f9fc-12be-4efa-8a74-ff0f50c1282b None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:30.865564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ac37cf-64bc-4e30-9b5a-fa5f6001887a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:30.866457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ac37cf-64bc-4e30-9b5a-fa5f6001887a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:30.866890 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ac37cf-64bc-4e30-9b5a-fa5f6001887a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:30.867355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ac37cf-64bc-4e30-9b5a-fa5f6001887a None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:30.868131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ac37cf-64bc-4e30-9b5a-fa5f6001887a None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:30.869844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ac37cf-64bc-4e30-9b5a-fa5f6001887a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:30.905104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-f1ac37cf-64bc-4e30-9b5a-fa5f6001887a None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 04 00:47:30.905104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:30.905104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:30.905104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:30.905104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:30.905104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:30.905104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:30.905104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:30.905104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 04 00:47:30.905104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:30.905104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:30.905104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:30.905104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:30.905104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:30.905104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:30.905104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:30.905104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:30.906361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:30.906361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:30.906361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:30.906361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:30.906361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:30.906361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:30.906361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:30.906361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:30.906361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:30.906361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:30.906361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:30.906361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:30.906361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:30.906361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:30.906361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:30.906361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:30.907372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:30.907372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:30.907372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:30.907372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:30.907372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:30.907372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:30.907372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:30.907372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:30.907372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:30.907372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:30.907372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:30.907372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:30.907372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 04 00:47:30.907372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:30.910320 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 380/768] 10.222.0.42 () {56 vars in 1129 bytes} [Sun Sep 4 00:47:30 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 371 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:30.915988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ac37cf-64bc-4e30-9b5a-fa5f6001887a None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:30.944976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2010f9fc-12be-4efa-8a74-ff0f50c1282b None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:30.980458 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21df7b73-bbc7-412b-98e0-6a7b3b26927d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:30.981315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21df7b73-bbc7-412b-98e0-6a7b3b26927d None admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:30.981696 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21df7b73-bbc7-412b-98e0-6a7b3b26927d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:30.982071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21df7b73-bbc7-412b-98e0-6a7b3b26927d None admin] PATH_INFO: `/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/57c88f16c7df40adb917f49804fda21f/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:30.984274 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21df7b73-bbc7-412b-98e0-6a7b3b26927d None admin] RBAC: Authorizing `identity:create_grant(project_id=b1f56314d44d4faaa88e735ea196783f, user_id=57c88f16c7df40adb917f49804fda21f, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:30.985918 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21df7b73-bbc7-412b-98e0-6a7b3b26927d None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:31.040178 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:31.060683 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 389/769] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:47:30 2022] PUT /identity/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/57c88f16c7df40adb917f49804fda21f/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:47:31.062331 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af274ea6-2356-4cfb-808c-67f36078e731 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:31.062691 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af274ea6-2356-4cfb-808c-67f36078e731 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:31.063036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af274ea6-2356-4cfb-808c-67f36078e731 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:31.063503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ac37cf-64bc-4e30-9b5a-fa5f6001887a None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:31.066219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 390/770] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:31.070944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392c0270-20ec-4ca3-8bf7-2d7fa5327ff7 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:31.071284 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392c0270-20ec-4ca3-8bf7-2d7fa5327ff7 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:31.071643 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392c0270-20ec-4ca3-8bf7-2d7fa5327ff7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:31.108427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8af4569-7ef1-4615-af00-619a75432790 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:31.109478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8af4569-7ef1-4615-af00-619a75432790 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:31.109933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8af4569-7ef1-4615-af00-619a75432790 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:31.110368 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8af4569-7ef1-4615-af00-619a75432790 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:31.111170 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8af4569-7ef1-4615-af00-619a75432790 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:31.111754 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8af4569-7ef1-4615-af00-619a75432790 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:31.113472 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8af4569-7ef1-4615-af00-619a75432790 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:31.114317 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-392c0270-20ec-4ca3-8bf7-2d7fa5327ff7 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:31.115625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-392c0270-20ec-4ca3-8bf7-2d7fa5327ff7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:31.119932 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 381/771] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Sep 4 00:47:30 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:31.125698 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8af4569-7ef1-4615-af00-619a75432790 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:31.203175 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:31.228873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 391/772] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:31 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:31.230620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c6c5ab-1d35-4abd-96fd-dfaf20af0896 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:31.231199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c6c5ab-1d35-4abd-96fd-dfaf20af0896 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:31.231734 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c6c5ab-1d35-4abd-96fd-dfaf20af0896 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:31.237628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 392/773] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:31.239347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61433f6-b93f-4408-8f02-27cf67ef536f None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:31.239825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61433f6-b93f-4408-8f02-27cf67ef536f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:31.240278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61433f6-b93f-4408-8f02-27cf67ef536f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:31.260264 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8af4569-7ef1-4615-af00-619a75432790 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:31.264691 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-c61433f6-b93f-4408-8f02-27cf67ef536f None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:31.266160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c61433f6-b93f-4408-8f02-27cf67ef536f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:31.302864 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5229f552-c757-4622-a33c-decb31506ca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:31.303575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5229f552-c757-4622-a33c-decb31506ca1 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:31.303860 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5229f552-c757-4622-a33c-decb31506ca1 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:31.304147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5229f552-c757-4622-a33c-decb31506ca1 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:31.305564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5229f552-c757-4622-a33c-decb31506ca1 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:31.305886 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5229f552-c757-4622-a33c-decb31506ca1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:31.307260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5229f552-c757-4622-a33c-decb31506ca1 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:31.313881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 382/774] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Sep 4 00:47:31 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:31.320309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5229f552-c757-4622-a33c-decb31506ca1 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:31.342067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:31.360737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 393/775] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:31 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:31.461842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5229f552-c757-4622-a33c-decb31506ca1 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:31.504432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50739ea-dec5-4393-96e7-25de22158686 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:31.505053 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50739ea-dec5-4393-96e7-25de22158686 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:31.505362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50739ea-dec5-4393-96e7-25de22158686 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:31.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50739ea-dec5-4393-96e7-25de22158686 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:31.506354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50739ea-dec5-4393-96e7-25de22158686 None admin] RBAC: Adding query filter params (interface=public, service_id=75d2cd2549374b5697a623f6a2c58592, region_id=RegionOne) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:31.506668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50739ea-dec5-4393-96e7-25de22158686 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:31.508029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50739ea-dec5-4393-96e7-25de22158686 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:31.520233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 383/776] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Sep 4 00:47:31 2022] GET /identity/v3/endpoints?service_id=75d2cd2549374b5697a623f6a2c58592&interface=public®ion_id=RegionOne => generated 194 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:31.525578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61433f6-b93f-4408-8f02-27cf67ef536f None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:31.552116 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfc45b6-71fc-4f6d-82b5-df1107a4ec7e None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:31.552452 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfc45b6-71fc-4f6d-82b5-df1107a4ec7e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:31.552741 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfc45b6-71fc-4f6d-82b5-df1107a4ec7e None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:31.554426 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 384/777] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:31.559978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6adedd6-de1d-4934-9a53-dc0432cf3dfc None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:31.560318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6adedd6-de1d-4934-9a53-dc0432cf3dfc None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:31.560597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6adedd6-de1d-4934-9a53-dc0432cf3dfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:31.589604 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-f6adedd6-de1d-4934-9a53-dc0432cf3dfc None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:31.591179 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f6adedd6-de1d-4934-9a53-dc0432cf3dfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:31.659709 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61433f6-b93f-4408-8f02-27cf67ef536f None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:31.666513 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 385/778] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:31 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:31.667712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6adedd6-de1d-4934-9a53-dc0432cf3dfc None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:31.671862 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:31.697862 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237d4a6d-623a-46b6-b322-057327870abc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:31.698543 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237d4a6d-623a-46b6-b322-057327870abc None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:31.698830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237d4a6d-623a-46b6-b322-057327870abc None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:31.699518 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237d4a6d-623a-46b6-b322-057327870abc None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:31.699767 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237d4a6d-623a-46b6-b322-057327870abc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:31.701274 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237d4a6d-623a-46b6-b322-057327870abc None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:31.704149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:31.709104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 394/779] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 4 00:47:31 2022] GET /identity/v3/services => generated 772 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:31.728001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237d4a6d-623a-46b6-b322-057327870abc None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:31.822861 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6adedd6-de1d-4934-9a53-dc0432cf3dfc None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:31.869598 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e53a40e-23a5-4cb3-bcf1-8972743c9b5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:31.870244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e53a40e-23a5-4cb3-bcf1-8972743c9b5e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:31.870521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e53a40e-23a5-4cb3-bcf1-8972743c9b5e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:31.870788 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e53a40e-23a5-4cb3-bcf1-8972743c9b5e None admin] PATH_INFO: `/v3/services/network` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:31.871385 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e53a40e-23a5-4cb3-bcf1-8972743c9b5e None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:31.872748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e53a40e-23a5-4cb3-bcf1-8972743c9b5e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:31.884239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-0e53a40e-23a5-4cb3-bcf1-8972743c9b5e None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 04 00:47:31.884239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:31.884239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:31.884239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:31.884239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:31.884239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:31.884239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:31.884239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:31.884239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 04 00:47:31.884239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:31.884239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:31.884239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:31.884239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:31.884239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:31.884239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:31.884239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:31.884239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:31.885651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:31.885651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:31.885651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:31.885651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:31.885651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:31.885651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:31.885651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:31.885651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:31.885651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:31.885651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:31.885651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:31.885651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:31.885651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:31.885651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:31.885651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:31.885651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:31.886858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:31.886858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:31.886858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:31.886858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:31.886858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:31.886858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:31.886858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:31.886858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:31.886858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:31.886858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:31.886858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:31.886858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:31.886858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 04 00:47:31.886858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:31.886858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 386/780] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:31 2022] GET /identity/v3/services/network => generated 88 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:31.893709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e53a40e-23a5-4cb3-bcf1-8972743c9b5e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:31.917923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237d4a6d-623a-46b6-b322-057327870abc None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:31.968551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0ea0ed-a9d0-4626-ac2d-44001e5210cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:31.970093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0ea0ed-a9d0-4626-ac2d-44001e5210cb None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:31.970608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0ea0ed-a9d0-4626-ac2d-44001e5210cb None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:31.971803 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c0ea0ed-a9d0-4626-ac2d-44001e5210cb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:31.980257 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:31.983522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c0ea0ed-a9d0-4626-ac2d-44001e5210cb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:31.989553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-9c0ea0ed-a9d0-4626-ac2d-44001e5210cb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:47:31.989553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:31.989553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:31.989553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:31.989553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:31.989553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:31.989553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:31.989553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:31.989553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:31.989553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:31.989553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:31.989553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:31.989553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:31.989553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:31.989553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:31.989553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:31.995763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:31.995763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:31.995763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:31.995763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:31.995763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:31.995763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:31.995763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:31.995763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:31.995763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:31.995763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:31.995763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:31.995763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:31.995763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:31.995763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:31.995763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:31.995763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:31.996742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:31.996742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:31.996742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:31.996742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:31.996742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:31.996742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:31.996742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:31.996742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:31.996742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:31.996742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:31.996742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:47:31.996742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:31.996742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 395/781] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:47:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:31.996742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0ea0ed-a9d0-4626-ac2d-44001e5210cb None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:32.028617 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e53a40e-23a5-4cb3-bcf1-8972743c9b5e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:32.037253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:32.073088 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01741a03-d4fd-4246-8a32-3a64b4838410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:32.073362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01741a03-d4fd-4246-8a32-3a64b4838410 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:32.073696 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01741a03-d4fd-4246-8a32-3a64b4838410 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:32.074028 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01741a03-d4fd-4246-8a32-3a64b4838410 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:32.074722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01741a03-d4fd-4246-8a32-3a64b4838410 None admin] RBAC: Adding query filter params (name=network) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:32.075105 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01741a03-d4fd-4246-8a32-3a64b4838410 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:32.076576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01741a03-d4fd-4246-8a32-3a64b4838410 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:32.081948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 387/782] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 4 00:47:31 2022] GET /identity/v3/services?name=network => generated 124 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:32.082966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01741a03-d4fd-4246-8a32-3a64b4838410 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:32.109969 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c0ea0ed-a9d0-4626-ac2d-44001e5210cb None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:32.198414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01741a03-d4fd-4246-8a32-3a64b4838410 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:32.231310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:32.237400 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fde58cf-f9d5-411a-b4bf-6d5328ecf746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:32.243628 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fde58cf-f9d5-411a-b4bf-6d5328ecf746 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:32.243628 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fde58cf-f9d5-411a-b4bf-6d5328ecf746 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:32.243628 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fde58cf-f9d5-411a-b4bf-6d5328ecf746 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:32.243628 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fde58cf-f9d5-411a-b4bf-6d5328ecf746 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:32.243628 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fde58cf-f9d5-411a-b4bf-6d5328ecf746 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:32.243628 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fde58cf-f9d5-411a-b4bf-6d5328ecf746 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:32.255929 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 388/783] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:47:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:32.257414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fde58cf-f9d5-411a-b4bf-6d5328ecf746 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:32.267507 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decad7ee-110e-4401-bd75-7f194c69325b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:32.269014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decad7ee-110e-4401-bd75-7f194c69325b None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:32.269584 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decad7ee-110e-4401-bd75-7f194c69325b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:32.270030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decad7ee-110e-4401-bd75-7f194c69325b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:32.282580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decad7ee-110e-4401-bd75-7f194c69325b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:32.289167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-decad7ee-110e-4401-bd75-7f194c69325b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:32.289167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:32.289167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:32.289167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:32.289167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:32.289167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:32.289167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:32.289167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:32.289167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:32.289167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:32.289167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:32.289167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:32.289167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:32.289167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:32.289167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:32.289167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:32.290097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:32.290097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:32.290097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:32.290097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:32.290097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:32.290097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:32.290097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:32.290097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:32.290097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:32.290097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:32.290097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:32.290097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:32.290097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:32.290097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:32.290097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:32.290097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:32.290913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:32.290913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:32.290913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:32.290913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:32.290913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:32.290913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:32.290913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:32.290913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:32.290913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:32.290913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:32.290913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:32.290913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:32.293708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 396/784] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:31 2022] GET /identity/v3/roles/service => generated 85 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:32.294661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decad7ee-110e-4401-bd75-7f194c69325b None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:32.375634 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fde58cf-f9d5-411a-b4bf-6d5328ecf746 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:32.402969 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:32.409992 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decad7ee-110e-4401-bd75-7f194c69325b None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:32.415453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe83fce0-9cd3-471b-86d4-685458af358b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:32.416194 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe83fce0-9cd3-471b-86d4-685458af358b None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:32.416556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe83fce0-9cd3-471b-86d4-685458af358b None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:32.416900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe83fce0-9cd3-471b-86d4-685458af358b None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:32.417631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe83fce0-9cd3-471b-86d4-685458af358b None admin] RBAC: Adding query filter params (type=network) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:32.417999 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe83fce0-9cd3-471b-86d4-685458af358b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:32.419431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe83fce0-9cd3-471b-86d4-685458af358b None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:32.430990 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 389/785] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 4 00:47:32 2022] GET /identity/v3/services?type=network => generated 350 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:32.432023 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe83fce0-9cd3-471b-86d4-685458af358b None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:32.448978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692aee07-4d06-4e04-b639-88813dc1d9b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:32.450054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692aee07-4d06-4e04-b639-88813dc1d9b2 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:32.450532 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692aee07-4d06-4e04-b639-88813dc1d9b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:32.451028 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692aee07-4d06-4e04-b639-88813dc1d9b2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:32.463516 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692aee07-4d06-4e04-b639-88813dc1d9b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:32.465455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692aee07-4d06-4e04-b639-88813dc1d9b2 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:32.484921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-692aee07-4d06-4e04-b639-88813dc1d9b2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:32.484921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:32.484921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:32.484921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:32.484921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:32.484921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:32.484921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:32.484921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:32.484921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:32.484921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:32.484921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:32.484921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:32.484921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:32.484921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:32.484921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:32.484921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:32.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:32.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:32.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:32.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:32.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:32.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:32.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:32.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:32.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:32.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:32.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:32.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:32.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:32.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:32.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:32.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:32.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:32.487699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:32.487699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:32.487699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:32.489714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 397/786] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:32.490647 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692aee07-4d06-4e04-b639-88813dc1d9b2 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:32.512013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:32.570351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe83fce0-9cd3-471b-86d4-685458af358b None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:32.608919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eec1197-2bce-4685-b91c-e02738b67e41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:32.609699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eec1197-2bce-4685-b91c-e02738b67e41 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:32.610076 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eec1197-2bce-4685-b91c-e02738b67e41 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:32.610484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eec1197-2bce-4685-b91c-e02738b67e41 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:32.611217 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eec1197-2bce-4685-b91c-e02738b67e41 None admin] RBAC: Adding query filter params (name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:32.611843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eec1197-2bce-4685-b91c-e02738b67e41 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:32.613381 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eec1197-2bce-4685-b91c-e02738b67e41 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:32.618865 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692aee07-4d06-4e04-b639-88813dc1d9b2 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:32.625457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 390/787] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:32.626502 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eec1197-2bce-4685-b91c-e02738b67e41 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:32.664426 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cf5ae8-26c4-4631-8ae4-e7163d5a5ef5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:32.666082 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cf5ae8-26c4-4631-8ae4-e7163d5a5ef5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:32.666538 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cf5ae8-26c4-4631-8ae4-e7163d5a5ef5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:32.666959 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cf5ae8-26c4-4631-8ae4-e7163d5a5ef5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:32.667798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cf5ae8-26c4-4631-8ae4-e7163d5a5ef5 None admin] RBAC: Adding query filter params (interface=public, service_id=144ec758e9ad40a583da2b33196492ec, region_id=RegionOne) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:32.668250 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cf5ae8-26c4-4631-8ae4-e7163d5a5ef5 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:32.669771 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cf5ae8-26c4-4631-8ae4-e7163d5a5ef5 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:32.680171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 398/788] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Sep 4 00:47:32 2022] GET /identity/v3/endpoints?service_id=144ec758e9ad40a583da2b33196492ec&interface=public®ion_id=RegionOne => generated 194 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:32.681678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cf5ae8-26c4-4631-8ae4-e7163d5a5ef5 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:32.698774 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:32.737407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eec1197-2bce-4685-b91c-e02738b67e41 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:32.757759 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:32.772154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1311b6bf-e0d5-464d-9b64-307f7b152bf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:32.772771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1311b6bf-e0d5-464d-9b64-307f7b152bf3 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:32.773053 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1311b6bf-e0d5-464d-9b64-307f7b152bf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:32.773351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1311b6bf-e0d5-464d-9b64-307f7b152bf3 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:32.774167 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1311b6bf-e0d5-464d-9b64-307f7b152bf3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:32.774391 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1311b6bf-e0d5-464d-9b64-307f7b152bf3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:32.775999 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1311b6bf-e0d5-464d-9b64-307f7b152bf3 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:32.788463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 391/789] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:32.789333 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1311b6bf-e0d5-464d-9b64-307f7b152bf3 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:32.801822 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cf5ae8-26c4-4631-8ae4-e7163d5a5ef5 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:32.849174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe59b36-6a04-4232-98f3-0a54f4630dcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:32.850362 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe59b36-6a04-4232-98f3-0a54f4630dcd None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:32.851166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe59b36-6a04-4232-98f3-0a54f4630dcd None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:32.851878 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe59b36-6a04-4232-98f3-0a54f4630dcd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:32.866879 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe59b36-6a04-4232-98f3-0a54f4630dcd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:32.868600 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe59b36-6a04-4232-98f3-0a54f4630dcd None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:32.888746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-1fe59b36-6a04-4232-98f3-0a54f4630dcd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:32.888746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:32.888746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:32.888746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:32.888746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:32.888746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:32.888746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:32.888746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:32.888746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:32.888746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:32.888746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:32.888746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:32.888746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:32.888746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:32.888746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:32.888746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:32.889750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:32.889750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:32.889750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:32.889750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:32.889750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:32.889750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:32.889750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:32.889750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:32.889750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:32.889750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:32.889750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:32.889750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:32.889750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:32.889750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:32.889750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:32.889750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:32.889750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:32.890639 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:32.890639 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:32.890639 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:32.890639 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 399/790] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:32.891230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe59b36-6a04-4232-98f3-0a54f4630dcd None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:32.907625 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1311b6bf-e0d5-464d-9b64-307f7b152bf3 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:32.952749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b18aad-8f96-4840-ba61-9d74560c2682 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:32.953596 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b18aad-8f96-4840-ba61-9d74560c2682 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:32.953981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b18aad-8f96-4840-ba61-9d74560c2682 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:32.954357 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b18aad-8f96-4840-ba61-9d74560c2682 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:32.955627 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b18aad-8f96-4840-ba61-9d74560c2682 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:32.957760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b18aad-8f96-4840-ba61-9d74560c2682 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:32.964507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 392/791] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 4 00:47:32 2022] GET /identity/v3/services => generated 772 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:32.965369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:32.973737 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b18aad-8f96-4840-ba61-9d74560c2682 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:33.046624 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe59b36-6a04-4232-98f3-0a54f4630dcd None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:33.079603 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:33.138665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b18aad-8f96-4840-ba61-9d74560c2682 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:33.139609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e682e9-61d8-452a-b520-62d916bf7089 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:33.140456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e682e9-61d8-452a-b520-62d916bf7089 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:33.140734 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e682e9-61d8-452a-b520-62d916bf7089 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:33.141995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e682e9-61d8-452a-b520-62d916bf7089 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:33.154356 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e682e9-61d8-452a-b520-62d916bf7089 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:33.156328 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e682e9-61d8-452a-b520-62d916bf7089 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:33.169602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-38e682e9-61d8-452a-b520-62d916bf7089 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 04 00:47:33.169602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:33.169602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:33.169602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:33.169602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:33.169602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:33.169602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:33.169602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:33.169602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:33.169602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:33.169602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:33.169602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:33.169602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:33.169602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:33.169602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:33.169602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:33.170590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:33.171506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:33.171506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:33.171506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:33.171506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:33.171506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:33.171506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:33.171506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:33.171506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:33.171506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 04 00:47:33.171506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:33.171506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 400/792] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:47:32 2022] GET /identity/v3/users/swift => generated 83 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:33.179949 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e682e9-61d8-452a-b520-62d916bf7089 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:33.194469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bc5013-64fe-46aa-ba0a-867c5651e7cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:33.195724 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bc5013-64fe-46aa-ba0a-867c5651e7cf None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:33.196505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bc5013-64fe-46aa-ba0a-867c5651e7cf None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:33.196985 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0bc5013-64fe-46aa-ba0a-867c5651e7cf None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:33.198031 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0bc5013-64fe-46aa-ba0a-867c5651e7cf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:33.198533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0bc5013-64fe-46aa-ba0a-867c5651e7cf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:33.200676 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0bc5013-64fe-46aa-ba0a-867c5651e7cf None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:33.213811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 393/793] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:33.222067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bc5013-64fe-46aa-ba0a-867c5651e7cf None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:33.348761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e682e9-61d8-452a-b520-62d916bf7089 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:33.388971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:33.403420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0bc5013-64fe-46aa-ba0a-867c5651e7cf None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:33.414140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3086543-1a9e-4431-9526-b445984fcf01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:33.414759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3086543-1a9e-4431-9526-b445984fcf01 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:33.415031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3086543-1a9e-4431-9526-b445984fcf01 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:33.415296 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3086543-1a9e-4431-9526-b445984fcf01 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:33.416025 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3086543-1a9e-4431-9526-b445984fcf01 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:33.416325 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3086543-1a9e-4431-9526-b445984fcf01 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:33.417791 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3086543-1a9e-4431-9526-b445984fcf01 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:33.419961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:33.459528 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 401/794] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Sep 4 00:47:33 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:33.460931 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8526f98d-9964-416b-bdbb-7ab123b583a9 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:33.461257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8526f98d-9964-416b-bdbb-7ab123b583a9 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:33.461532 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8526f98d-9964-416b-bdbb-7ab123b583a9 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:33.462970 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 402/795] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:33.468970 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8526f98d-9964-416b-bdbb-7ab123b583a9 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:33.480881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1142228b-0e47-47f8-81b5-5ba522f3bc03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:33.481952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1142228b-0e47-47f8-81b5-5ba522f3bc03 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:33.482428 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1142228b-0e47-47f8-81b5-5ba522f3bc03 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:33.482965 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1142228b-0e47-47f8-81b5-5ba522f3bc03 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:33.490609 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1142228b-0e47-47f8-81b5-5ba522f3bc03 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:33.492206 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1142228b-0e47-47f8-81b5-5ba522f3bc03 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:33.504995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-1142228b-0e47-47f8-81b5-5ba522f3bc03 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 04 00:47:33.504995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:33.504995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:33.504995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:33.504995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:33.504995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:33.504995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:33.504995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:33.504995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:33.504995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:33.504995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:33.504995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:33.504995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:33.504995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:33.504995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:33.504995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:33.505981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:33.506916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:33.506916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:33.506916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:33.506916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:33.506916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:33.506916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:33.506916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:33.506916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:33.506916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 04 00:47:33.506916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:33.507978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 394/796] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 4 00:47:33 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:33.509864 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d009e203-4b24-4e58-ab2c-23a84bc4baa0 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:33.510350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d009e203-4b24-4e58-ab2c-23a84bc4baa0 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:33.510640 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d009e203-4b24-4e58-ab2c-23a84bc4baa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:33.530150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-d009e203-4b24-4e58-ab2c-23a84bc4baa0 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:33.531351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d009e203-4b24-4e58-ab2c-23a84bc4baa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:33.603255 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8526f98d-9964-416b-bdbb-7ab123b583a9 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:33.611541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 395/797] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:33 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:33.618133 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d009e203-4b24-4e58-ab2c-23a84bc4baa0 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:33.644454 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1ae8bb-ea8a-418d-9a7a-923e0f6428a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:33.645382 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1ae8bb-ea8a-418d-9a7a-923e0f6428a5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:33.645651 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1ae8bb-ea8a-418d-9a7a-923e0f6428a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:33.645928 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee1ae8bb-ea8a-418d-9a7a-923e0f6428a5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:33.658511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1ae8bb-ea8a-418d-9a7a-923e0f6428a5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:33.661000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee1ae8bb-ea8a-418d-9a7a-923e0f6428a5 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:33.679273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-ee1ae8bb-ea8a-418d-9a7a-923e0f6428a5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:33.679273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:33.679273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:33.679273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:33.679273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:33.679273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:33.679273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:33.679273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:33.679273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:33.679273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:33.679273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:33.679273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:33.679273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:33.679273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:33.679273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:33.679273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:33.680437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:33.680437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:33.680437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:33.680437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:33.680437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:33.680437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:33.680437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:33.680437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:33.680437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:33.680437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:33.680437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:33.680437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:33.680437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:33.680437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:33.680437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:33.680437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:33.680437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:33.681499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:33.681499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:33.681499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:33.681499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 403/798] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:33.686543 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1ae8bb-ea8a-418d-9a7a-923e0f6428a5 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:33.749164 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:33.795365 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:33.806229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d009e203-4b24-4e58-ab2c-23a84bc4baa0 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:33.868728 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee1ae8bb-ea8a-418d-9a7a-923e0f6428a5 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:33.870586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9742ed86-8169-41c7-a82a-566fe984930f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:33.871194 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9742ed86-8169-41c7-a82a-566fe984930f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:33.871468 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9742ed86-8169-41c7-a82a-566fe984930f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:33.871738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9742ed86-8169-41c7-a82a-566fe984930f None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:33.872468 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9742ed86-8169-41c7-a82a-566fe984930f None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:33.872831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9742ed86-8169-41c7-a82a-566fe984930f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:33.874786 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9742ed86-8169-41c7-a82a-566fe984930f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:33.898266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 396/799] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Sep 4 00:47:33 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:33.903699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9742ed86-8169-41c7-a82a-566fe984930f None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:33.954405 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9df6137-939e-4142-b107-9263c963be89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:33.954932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9df6137-939e-4142-b107-9263c963be89 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:33.955258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9df6137-939e-4142-b107-9263c963be89 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:33.955523 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9df6137-939e-4142-b107-9263c963be89 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:33.956275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9df6137-939e-4142-b107-9263c963be89 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:33.956570 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9df6137-939e-4142-b107-9263c963be89 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:33.957967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9df6137-939e-4142-b107-9263c963be89 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:33.969687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 404/800] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:33.971194 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a01b361-315f-42fa-8d13-1d8c4efe4446 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:33.971493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a01b361-315f-42fa-8d13-1d8c4efe4446 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:33.971765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a01b361-315f-42fa-8d13-1d8c4efe4446 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:33.973242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 405/801] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:33.974836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a01b361-315f-42fa-8d13-1d8c4efe4446 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:34.027784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9742ed86-8169-41c7-a82a-566fe984930f None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:34.064249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1b6064-bd97-4116-85bb-274387d33745 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:34.065425 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1b6064-bd97-4116-85bb-274387d33745 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:34.065706 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1b6064-bd97-4116-85bb-274387d33745 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:34.066086 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1b6064-bd97-4116-85bb-274387d33745 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:34.076395 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1b6064-bd97-4116-85bb-274387d33745 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:34.077445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:34.077871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1b6064-bd97-4116-85bb-274387d33745 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:34.088586 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a01b361-315f-42fa-8d13-1d8c4efe4446 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:34.093881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-da1b6064-bd97-4116-85bb-274387d33745 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:34.093881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:34.093881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:34.093881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:34.093881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:34.093881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:34.093881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:34.093881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:34.093881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:34.093881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:34.093881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:34.093881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:34.093881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:34.093881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:34.093881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:34.093881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:34.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:34.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:34.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:34.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:34.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:34.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:34.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:34.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:34.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:34.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:34.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:34.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:34.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:34.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:34.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:34.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:34.094782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:34.101241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:34.101241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:34.101241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:34.101241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 397/802] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:34.101241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdc615a-1ee2-4576-9c5f-ef79b9490910 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:34.101241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdc615a-1ee2-4576-9c5f-ef79b9490910 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:34.101241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cdc615a-1ee2-4576-9c5f-ef79b9490910 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:34.109156 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:34.119769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-7cdc615a-1ee2-4576-9c5f-ef79b9490910 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:34.120961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-7cdc615a-1ee2-4576-9c5f-ef79b9490910 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:34.132255 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0035d557-3708-4ae7-984b-d71ab493eb5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:34.132927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0035d557-3708-4ae7-984b-d71ab493eb5c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:34.133264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0035d557-3708-4ae7-984b-d71ab493eb5c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:34.133555 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0035d557-3708-4ae7-984b-d71ab493eb5c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:34.140235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0035d557-3708-4ae7-984b-d71ab493eb5c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:34.141712 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0035d557-3708-4ae7-984b-d71ab493eb5c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:34.153630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-0035d557-3708-4ae7-984b-d71ab493eb5c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:34.153630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:34.153630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:34.153630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:34.153630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:34.153630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:34.153630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:34.153630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:34.153630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:34.153630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:34.153630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:34.153630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:34.153630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:34.153630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:34.153630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:34.154742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:34.155827 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:34.155827 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:34.155827 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:34.155827 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:34.155827 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:34.155827 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:34.155827 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:34.155827 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:34.160003 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 406/803] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:34.160994 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0035d557-3708-4ae7-984b-d71ab493eb5c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:34.197973 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 398/804] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:34 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:34.199066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdc615a-1ee2-4576-9c5f-ef79b9490910 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:34.283857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0035d557-3708-4ae7-984b-d71ab493eb5c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:34.318537 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cdc615a-1ee2-4576-9c5f-ef79b9490910 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:34.319569 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c50f3e9-2a73-48d1-8299-a110e46db045 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:34.320203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c50f3e9-2a73-48d1-8299-a110e46db045 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:34.320481 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c50f3e9-2a73-48d1-8299-a110e46db045 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:34.320751 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c50f3e9-2a73-48d1-8299-a110e46db045 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:34.331963 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c50f3e9-2a73-48d1-8299-a110e46db045 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:34.337199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-9c50f3e9-2a73-48d1-8299-a110e46db045 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:47:34.337199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:34.337199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:34.337199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:34.337199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:34.337199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:34.337199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:34.337199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:34.337199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:34.337199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:34.337199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:34.337199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:34.337199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:34.337199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:34.337199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:34.337199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:34.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:34.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:34.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:34.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:34.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:34.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:34.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:34.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:34.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:34.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:34.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:34.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:34.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:34.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:34.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:34.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:34.339553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:34.339553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:34.339553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:34.339553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:34.339553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:34.339553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:34.339553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:34.339553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:34.339553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:34.339553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:34.339553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:47:34.339553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:34.339553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 407/805] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:47:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:34.339553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c50f3e9-2a73-48d1-8299-a110e46db045 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:34.355303 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:34.362182 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1878c78-b862-4f25-bb86-84d4cee03f71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:34.363036 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1878c78-b862-4f25-bb86-84d4cee03f71 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:34.363534 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1878c78-b862-4f25-bb86-84d4cee03f71 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:34.364002 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1878c78-b862-4f25-bb86-84d4cee03f71 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:34.364866 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1878c78-b862-4f25-bb86-84d4cee03f71 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:34.365403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1878c78-b862-4f25-bb86-84d4cee03f71 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:34.367009 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1878c78-b862-4f25-bb86-84d4cee03f71 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:34.368645 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:34.380360 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 399/806] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:34.381561 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1878c78-b862-4f25-bb86-84d4cee03f71 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:34.447536 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c50f3e9-2a73-48d1-8299-a110e46db045 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:34.486753 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402c03eb-15fd-4f1f-a2a9-c598269567a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:34.487426 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402c03eb-15fd-4f1f-a2a9-c598269567a5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:34.487724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402c03eb-15fd-4f1f-a2a9-c598269567a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:34.488033 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402c03eb-15fd-4f1f-a2a9-c598269567a5 None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:34.488736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402c03eb-15fd-4f1f-a2a9-c598269567a5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:34.489041 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402c03eb-15fd-4f1f-a2a9-c598269567a5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:34.490499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402c03eb-15fd-4f1f-a2a9-c598269567a5 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:34.491334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1878c78-b862-4f25-bb86-84d4cee03f71 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:34.505125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 408/807] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:34.505125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402c03eb-15fd-4f1f-a2a9-c598269567a5 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:34.528418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e649b2-72d1-4011-a6e8-dfa8737496e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:34.529100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e649b2-72d1-4011-a6e8-dfa8737496e4 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:34.529397 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e649b2-72d1-4011-a6e8-dfa8737496e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:34.529669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e649b2-72d1-4011-a6e8-dfa8737496e4 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:34.530336 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e649b2-72d1-4011-a6e8-dfa8737496e4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:34.530644 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e649b2-72d1-4011-a6e8-dfa8737496e4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:34.531992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e649b2-72d1-4011-a6e8-dfa8737496e4 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:34.545578 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 400/808] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:47:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:34.545578 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e649b2-72d1-4011-a6e8-dfa8737496e4 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:34.588400 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:34.606292 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:34.621316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402c03eb-15fd-4f1f-a2a9-c598269567a5 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:34.651526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e649b2-72d1-4011-a6e8-dfa8737496e4 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:34.656794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7693a19-21d6-44f9-a768-1b2bdbfa3eb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:34.657456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7693a19-21d6-44f9-a768-1b2bdbfa3eb4 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:34.657732 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7693a19-21d6-44f9-a768-1b2bdbfa3eb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:34.657997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7693a19-21d6-44f9-a768-1b2bdbfa3eb4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:34.666830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7693a19-21d6-44f9-a768-1b2bdbfa3eb4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:34.668275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7693a19-21d6-44f9-a768-1b2bdbfa3eb4 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:34.679699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-e7693a19-21d6-44f9-a768-1b2bdbfa3eb4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:34.679699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:34.679699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:34.679699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:34.679699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:34.679699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:34.679699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:34.679699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:34.679699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:34.679699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:34.679699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:34.679699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:34.679699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:34.679699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:34.679699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:34.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:34.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:34.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:34.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:34.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:34.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:34.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:34.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:34.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:34.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:34.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:34.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:34.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:34.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:34.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:34.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:34.680752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:34.681738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:34.681738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:34.681738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:34.681738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:34.681738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:34.681738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:34.681738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:34.681738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:34.681738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 409/809] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:34.681738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7693a19-21d6-44f9-a768-1b2bdbfa3eb4 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:34.694087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a174fd0-e926-4d1d-8354-032aad1b5f96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:34.695097 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a174fd0-e926-4d1d-8354-032aad1b5f96 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:34.695574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a174fd0-e926-4d1d-8354-032aad1b5f96 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:34.696172 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a174fd0-e926-4d1d-8354-032aad1b5f96 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:34.696835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a174fd0-e926-4d1d-8354-032aad1b5f96 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:34.698268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a174fd0-e926-4d1d-8354-032aad1b5f96 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:34.708666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-8a174fd0-e926-4d1d-8354-032aad1b5f96 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 04 00:47:34.708666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:34.708666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:34.708666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:34.708666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:34.708666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:34.708666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:34.708666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:34.708666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 04 00:47:34.708666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:34.708666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:34.708666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:34.708666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:34.708666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:34.708666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:34.708666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:34.708666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:34.709695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:34.709695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:34.709695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:34.709695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:34.709695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:34.709695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:34.709695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:34.709695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:34.709695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:34.709695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:34.709695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:34.709695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:34.709695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:34.709695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:34.709695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:34.709695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:34.710544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:34.710544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:34.710544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:34.710544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:34.710544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:34.710544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:34.710544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:34.710544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:34.710544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:34.710544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:34.710544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:34.710544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:34.710544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 04 00:47:34.710544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:34.710544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 401/810] 10.222.0.42 () {56 vars in 1129 bytes} [Sun Sep 4 00:47:34 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:34.711362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a174fd0-e926-4d1d-8354-032aad1b5f96 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:34.798106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7693a19-21d6-44f9-a768-1b2bdbfa3eb4 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:34.824767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a174fd0-e926-4d1d-8354-032aad1b5f96 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:34.826027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:34.832281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bfded1-fcd0-4e22-8162-d8385dd684d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:34.832936 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bfded1-fcd0-4e22-8162-d8385dd684d3 None admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:34.833247 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bfded1-fcd0-4e22-8162-d8385dd684d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:34.842663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bfded1-fcd0-4e22-8162-d8385dd684d3 None admin] PATH_INFO: `/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/123ab6ff796c4e599ce1d6f4095bd898/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:34.842663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9bfded1-fcd0-4e22-8162-d8385dd684d3 None admin] RBAC: Authorizing `identity:create_grant(project_id=b1f56314d44d4faaa88e735ea196783f, user_id=123ab6ff796c4e599ce1d6f4095bd898, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:34.842663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9bfded1-fcd0-4e22-8162-d8385dd684d3 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:34.847287 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:34.860554 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6aef06-7729-4bd1-9cfe-8f3ec954fd6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:34.865600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6aef06-7729-4bd1-9cfe-8f3ec954fd6f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:34.865600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6aef06-7729-4bd1-9cfe-8f3ec954fd6f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:34.865600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6aef06-7729-4bd1-9cfe-8f3ec954fd6f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:34.871865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6aef06-7729-4bd1-9cfe-8f3ec954fd6f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:34.873586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6aef06-7729-4bd1-9cfe-8f3ec954fd6f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:34.887823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-4f6aef06-7729-4bd1-9cfe-8f3ec954fd6f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:34.887823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:34.887823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:34.887823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:34.887823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:34.887823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:34.887823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:34.887823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:34.887823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:34.887823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:34.887823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:34.887823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:34.887823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:34.887823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:34.887823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:34.887823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:34.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:34.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:34.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:34.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:34.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:34.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:34.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:34.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:34.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:34.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:34.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:34.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:34.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:34.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:34.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:34.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:34.888921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:34.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:34.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:34.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:34.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 402/811] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:34.890246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6aef06-7729-4bd1-9cfe-8f3ec954fd6f None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:34.929464 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 410/812] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:47:34 2022] PUT /identity/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/123ab6ff796c4e599ce1d6f4095bd898/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:47:34.930435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bfded1-fcd0-4e22-8162-d8385dd684d3 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:35.039501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6aef06-7729-4bd1-9cfe-8f3ec954fd6f None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:35.055897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bfded1-fcd0-4e22-8162-d8385dd684d3 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:35.083089 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e850a1-28e8-4a1b-a557-834e2ad0de03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:35.084013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e850a1-28e8-4a1b-a557-834e2ad0de03 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:35.084301 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e850a1-28e8-4a1b-a557-834e2ad0de03 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:35.084599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e850a1-28e8-4a1b-a557-834e2ad0de03 None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:35.085520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e850a1-28e8-4a1b-a557-834e2ad0de03 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:35.085633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e850a1-28e8-4a1b-a557-834e2ad0de03 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:35.087071 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e850a1-28e8-4a1b-a557-834e2ad0de03 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:35.097203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacecef3-c30c-456b-b665-e63af6075377 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:35.097859 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacecef3-c30c-456b-b665-e63af6075377 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:35.098143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacecef3-c30c-456b-b665-e63af6075377 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:35.098419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacecef3-c30c-456b-b665-e63af6075377 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:35.099075 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacecef3-c30c-456b-b665-e63af6075377 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:35.099380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacecef3-c30c-456b-b665-e63af6075377 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:35.102886 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 403/813] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:35.103021 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacecef3-c30c-456b-b665-e63af6075377 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:35.103098 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e850a1-28e8-4a1b-a557-834e2ad0de03 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:35.111119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 411/814] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Sep 4 00:47:34 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:35.112750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c7c816-6bf9-4923-84ec-2b6ea66b8572 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:35.113289 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c7c816-6bf9-4923-84ec-2b6ea66b8572 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:35.113565 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c7c816-6bf9-4923-84ec-2b6ea66b8572 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:35.114901 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 412/815] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:35.115431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c7c816-6bf9-4923-84ec-2b6ea66b8572 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:35.137656 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:35.178614 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:35.213946 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e850a1-28e8-4a1b-a557-834e2ad0de03 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:35.226074 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c7c816-6bf9-4923-84ec-2b6ea66b8572 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:35.252754 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a87acf-91ce-45a4-b2c3-e6e18a3aa9b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:35.253646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a87acf-91ce-45a4-b2c3-e6e18a3aa9b4 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:35.254069 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a87acf-91ce-45a4-b2c3-e6e18a3aa9b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:35.254485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a87acf-91ce-45a4-b2c3-e6e18a3aa9b4 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:35.255268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a87acf-91ce-45a4-b2c3-e6e18a3aa9b4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:35.255713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a87acf-91ce-45a4-b2c3-e6e18a3aa9b4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:35.257256 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a87acf-91ce-45a4-b2c3-e6e18a3aa9b4 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:35.262031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049eaa1e-8af8-4b84-a3b5-ca3f8dc47483 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:35.262689 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049eaa1e-8af8-4b84-a3b5-ca3f8dc47483 None admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:35.262972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049eaa1e-8af8-4b84-a3b5-ca3f8dc47483 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:35.263246 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049eaa1e-8af8-4b84-a3b5-ca3f8dc47483 None admin] PATH_INFO: `/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/70cef7c271e046a3b49230057cc2d28f/roles/df912a77fbaa479db172f76a214f8bc8` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:35.274802 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 404/816] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:35.274802 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a87acf-91ce-45a4-b2c3-e6e18a3aa9b4 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:35.281738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049eaa1e-8af8-4b84-a3b5-ca3f8dc47483 None admin] RBAC: Authorizing `identity:create_grant(project_id=b1f56314d44d4faaa88e735ea196783f, user_id=70cef7c271e046a3b49230057cc2d28f, role_id=df912a77fbaa479db172f76a214f8bc8)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:35.283218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049eaa1e-8af8-4b84-a3b5-ca3f8dc47483 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:35.353566 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 413/817] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:47:35 2022] PUT /identity/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/70cef7c271e046a3b49230057cc2d28f/roles/df912a77fbaa479db172f76a214f8bc8 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:47:35.354758 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8289dd2b-2c4a-4d0f-a196-a32f23a6ba90 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:35.355044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8289dd2b-2c4a-4d0f-a196-a32f23a6ba90 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:35.355318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8289dd2b-2c4a-4d0f-a196-a32f23a6ba90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:35.378169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:35.379286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-8289dd2b-2c4a-4d0f-a196-a32f23a6ba90 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:35.380697 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-8289dd2b-2c4a-4d0f-a196-a32f23a6ba90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:35.390311 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a87acf-91ce-45a4-b2c3-e6e18a3aa9b4 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:35.430956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a1b87b-298e-4bd2-9dcb-e1e7f478faf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:35.431663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a1b87b-298e-4bd2-9dcb-e1e7f478faf8 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:35.432012 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a1b87b-298e-4bd2-9dcb-e1e7f478faf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:35.432372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a1b87b-298e-4bd2-9dcb-e1e7f478faf8 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:35.433099 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a1b87b-298e-4bd2-9dcb-e1e7f478faf8 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:35.433492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a1b87b-298e-4bd2-9dcb-e1e7f478faf8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:35.434998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a1b87b-298e-4bd2-9dcb-e1e7f478faf8 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:35.443582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 405/818] 10.222.0.42 () {56 vars in 1124 bytes} [Sun Sep 4 00:47:35 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:35.444460 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a1b87b-298e-4bd2-9dcb-e1e7f478faf8 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:35.482625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 414/819] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:35 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:35.483648 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8289dd2b-2c4a-4d0f-a196-a32f23a6ba90 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:35.526701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:35.566109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a1b87b-298e-4bd2-9dcb-e1e7f478faf8 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:35.605593 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8ae959-54d9-45e4-8410-cbded18e01c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:35.605945 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8289dd2b-2c4a-4d0f-a196-a32f23a6ba90 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:35.606138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8ae959-54d9-45e4-8410-cbded18e01c9 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:35.606418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8ae959-54d9-45e4-8410-cbded18e01c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:35.606690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8ae959-54d9-45e4-8410-cbded18e01c9 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:35.618287 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8ae959-54d9-45e4-8410-cbded18e01c9 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:35.619908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8ae959-54d9-45e4-8410-cbded18e01c9 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:35.633213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-2c8ae959-54d9-45e4-8410-cbded18e01c9 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:47:35.633213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:35.633213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:35.633213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:35.633213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:35.633213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:35.633213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:35.633213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:35.633213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:35.633213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:35.633213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:35.633213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:35.633213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:35.633213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:35.633213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:35.633213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:35.634251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:35.635169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:35.635169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:35.635169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:35.635169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:35.635169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:35.635169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:35.635169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:35.635169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:35.635169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:47:35.635169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:35.637337 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 406/820] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 4 00:47:35 2022] GET /identity/v3/users/nova => generated 82 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:35.640669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8ae959-54d9-45e4-8410-cbded18e01c9 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:35.652317 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c4700a-5e21-4d22-942d-23da6e1f2f33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:35.653365 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c4700a-5e21-4d22-942d-23da6e1f2f33 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:35.653848 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c4700a-5e21-4d22-942d-23da6e1f2f33 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:35.654439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c4700a-5e21-4d22-942d-23da6e1f2f33 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:35.655300 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c4700a-5e21-4d22-942d-23da6e1f2f33 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:35.656960 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c4700a-5e21-4d22-942d-23da6e1f2f33 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:35.661159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:35.759612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8ae959-54d9-45e4-8410-cbded18e01c9 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:35.797007 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e66f1-2aef-4fe3-92c9-2724ff68adf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:35.799824 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e66f1-2aef-4fe3-92c9-2724ff68adf9 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:35.799824 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e66f1-2aef-4fe3-92c9-2724ff68adf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:35.799824 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e66f1-2aef-4fe3-92c9-2724ff68adf9 None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:35.799824 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4e66f1-2aef-4fe3-92c9-2724ff68adf9 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:35.799824 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4e66f1-2aef-4fe3-92c9-2724ff68adf9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:35.800971 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4e66f1-2aef-4fe3-92c9-2724ff68adf9 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:35.805194 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 415/821] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 00:47:35 2022] POST /identity/v3/endpoints => generated 346 bytes in 322 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:47:35.824301 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 407/822] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 4 00:47:35 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:35.825656 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c4700a-5e21-4d22-942d-23da6e1f2f33 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:35.827533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e66f1-2aef-4fe3-92c9-2724ff68adf9 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:35.927367 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:35.959217 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e66f1-2aef-4fe3-92c9-2724ff68adf9 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:36.007137 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:36.013583 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f49740-3cb5-457a-873c-b463a59cc0b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:36.014758 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f49740-3cb5-457a-873c-b463a59cc0b7 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:36.015044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f49740-3cb5-457a-873c-b463a59cc0b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:36.015317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f49740-3cb5-457a-873c-b463a59cc0b7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:36.018684 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c4700a-5e21-4d22-942d-23da6e1f2f33 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:36.059314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e2f5c9-c10c-452a-9237-dcc6c6962dd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:36.059937 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e2f5c9-c10c-452a-9237-dcc6c6962dd2 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:36.060216 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e2f5c9-c10c-452a-9237-dcc6c6962dd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:36.060484 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e2f5c9-c10c-452a-9237-dcc6c6962dd2 None admin] PATH_INFO: `/v3/services/network` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:36.061069 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e2f5c9-c10c-452a-9237-dcc6c6962dd2 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:36.062685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e2f5c9-c10c-452a-9237-dcc6c6962dd2 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:36.073444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-c1e2f5c9-c10c-452a-9237-dcc6c6962dd2 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 04 00:47:36.073444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:36.073444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:36.073444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:36.073444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:36.073444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:36.073444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:36.073444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:36.073444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 04 00:47:36.073444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:36.073444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:36.073444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:36.073444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:36.073444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:36.073444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:36.073444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:36.073444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:36.075040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:36.075040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:36.075040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:36.075040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:36.075040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:36.075040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:36.075040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:36.075040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:36.075040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:36.075040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:36.075040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:36.075040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:36.075040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:36.075040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:36.075040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:36.075040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:36.078158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:36.078158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:36.078158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:36.078158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:36.078158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:36.078158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:36.078158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:36.078158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:36.078158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:36.078158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:36.078158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:36.078158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:36.078158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 04 00:47:36.078158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:36.078158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 416/823] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:35 2022] GET /identity/v3/services/network => generated 88 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:36.083145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e2f5c9-c10c-452a-9237-dcc6c6962dd2 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:36.111697 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f49740-3cb5-457a-873c-b463a59cc0b7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:36.115604 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f49740-3cb5-457a-873c-b463a59cc0b7 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:36.206281 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:36.227486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e2f5c9-c10c-452a-9237-dcc6c6962dd2 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:36.269561 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64c0ad3-b7a1-4410-9800-c578e8f3fee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:36.270316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64c0ad3-b7a1-4410-9800-c578e8f3fee9 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:36.270603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64c0ad3-b7a1-4410-9800-c578e8f3fee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:36.270876 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64c0ad3-b7a1-4410-9800-c578e8f3fee9 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:36.271532 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64c0ad3-b7a1-4410-9800-c578e8f3fee9 None admin] RBAC: Adding query filter params (name=network) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:36.271839 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64c0ad3-b7a1-4410-9800-c578e8f3fee9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:36.273363 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64c0ad3-b7a1-4410-9800-c578e8f3fee9 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:36.280314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 417/824] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 4 00:47:36 2022] GET /identity/v3/services?name=network => generated 124 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:36.288512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64c0ad3-b7a1-4410-9800-c578e8f3fee9 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:36.292246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 408/825] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:35 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 465 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 04 00:47:36.299223 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f49740-3cb5-457a-873c-b463a59cc0b7 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:36.338344 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:36.436392 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64c0ad3-b7a1-4410-9800-c578e8f3fee9 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:36.475229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f49740-3cb5-457a-873c-b463a59cc0b7 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:36.504770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b3d830-b4d0-4d6a-94ac-4ea1618641c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:36.505427 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b3d830-b4d0-4d6a-94ac-4ea1618641c6 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:36.505708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b3d830-b4d0-4d6a-94ac-4ea1618641c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:36.505984 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b3d830-b4d0-4d6a-94ac-4ea1618641c6 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:36.506625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b3d830-b4d0-4d6a-94ac-4ea1618641c6 None admin] RBAC: Adding query filter params (type=network) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:36.506930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b3d830-b4d0-4d6a-94ac-4ea1618641c6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:36.508302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b3d830-b4d0-4d6a-94ac-4ea1618641c6 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:36.514002 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 418/826] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 4 00:47:36 2022] GET /identity/v3/services?type=network => generated 350 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:36.519915 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2fafe3-dda5-4383-9a47-96f92b6388a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:36.520562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2fafe3-dda5-4383-9a47-96f92b6388a6 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:36.520848 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2fafe3-dda5-4383-9a47-96f92b6388a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:36.521150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2fafe3-dda5-4383-9a47-96f92b6388a6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:36.526011 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b3d830-b4d0-4d6a-94ac-4ea1618641c6 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:36.532607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2fafe3-dda5-4383-9a47-96f92b6388a6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:36.535219 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2fafe3-dda5-4383-9a47-96f92b6388a6 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:36.552510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-6f2fafe3-dda5-4383-9a47-96f92b6388a6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:36.552510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:36.552510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:36.552510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:36.552510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:36.552510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:36.552510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:36.552510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:36.552510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:36.552510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:36.552510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:36.552510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:36.552510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:36.552510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:36.552510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:36.552510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:36.553530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:36.553530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:36.553530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:36.553530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:36.553530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:36.553530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:36.553530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:36.553530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:36.553530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:36.553530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:36.553530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:36.553530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:36.553530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:36.553530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:36.553530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:36.553530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:36.553530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:36.554420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:36.554420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:36.554420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:36.554420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 409/827] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:36.564617 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2fafe3-dda5-4383-9a47-96f92b6388a6 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:36.615458 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:36.671281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:36.695557 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b3d830-b4d0-4d6a-94ac-4ea1618641c6 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:36.757241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc13be9b-b9bc-4a8c-9c32-f11b349e4806 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:36.757777 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc13be9b-b9bc-4a8c-9c32-f11b349e4806 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:36.758057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc13be9b-b9bc-4a8c-9c32-f11b349e4806 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:36.758330 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc13be9b-b9bc-4a8c-9c32-f11b349e4806 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:36.759003 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc13be9b-b9bc-4a8c-9c32-f11b349e4806 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:36.761643 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc13be9b-b9bc-4a8c-9c32-f11b349e4806 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:36.770008 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2fafe3-dda5-4383-9a47-96f92b6388a6 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:36.808474 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786dbaf3-0241-47a2-ac7f-858145c3e4bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:36.809122 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786dbaf3-0241-47a2-ac7f-858145c3e4bb None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:36.809456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786dbaf3-0241-47a2-ac7f-858145c3e4bb None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:36.809746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786dbaf3-0241-47a2-ac7f-858145c3e4bb None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:36.810428 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786dbaf3-0241-47a2-ac7f-858145c3e4bb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:36.810808 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786dbaf3-0241-47a2-ac7f-858145c3e4bb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:36.812239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786dbaf3-0241-47a2-ac7f-858145c3e4bb None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:36.830798 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 410/828] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:36.832496 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786dbaf3-0241-47a2-ac7f-858145c3e4bb None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:36.863690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 419/829] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 00:47:36 2022] POST /identity/v3/endpoints => generated 337 bytes in 338 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:47:36.952939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:36.979398 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786dbaf3-0241-47a2-ac7f-858145c3e4bb None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:37.040328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34547a7d-14c4-4d1f-a6d8-bdd8968423e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:37.040882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34547a7d-14c4-4d1f-a6d8-bdd8968423e3 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:37.041196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34547a7d-14c4-4d1f-a6d8-bdd8968423e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:37.041472 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34547a7d-14c4-4d1f-a6d8-bdd8968423e3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:37.050087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34547a7d-14c4-4d1f-a6d8-bdd8968423e3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:37.051532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34547a7d-14c4-4d1f-a6d8-bdd8968423e3 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:37.067456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-34547a7d-14c4-4d1f-a6d8-bdd8968423e3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:37.067456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:37.067456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:37.067456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:37.067456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:37.067456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:37.067456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:37.067456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:37.067456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:37.067456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:37.067456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:37.067456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:37.067456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:37.067456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:37.067456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:37.068477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:37.068477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:37.068477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:37.068477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:37.068477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:37.068477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:37.068477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:37.068477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:37.068477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:37.068477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:37.068477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:37.068477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:37.068477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:37.068477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:37.068477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:37.068477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:37.068477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:37.069401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:37.069401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:37.069401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:37.069401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:37.069401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:37.069401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:37.069401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:37.069401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:37.073212 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 411/830] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:37.074917 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc13be9b-b9bc-4a8c-9c32-f11b349e4806 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:37.183949 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc13be9b-b9bc-4a8c-9c32-f11b349e4806 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:37.223383 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30769c0-9511-42cb-8b45-f0e6add18770 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:37.224007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30769c0-9511-42cb-8b45-f0e6add18770 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:37.224287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30769c0-9511-42cb-8b45-f0e6add18770 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:37.224562 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30769c0-9511-42cb-8b45-f0e6add18770 None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:37.225294 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30769c0-9511-42cb-8b45-f0e6add18770 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:37.225606 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30769c0-9511-42cb-8b45-f0e6add18770 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:37.227047 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30769c0-9511-42cb-8b45-f0e6add18770 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:37.234107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2b3e66-3263-4c40-958f-631d581881b8 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:37.234397 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2b3e66-3263-4c40-958f-631d581881b8 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:37.234672 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2b3e66-3263-4c40-958f-631d581881b8 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:37.236002 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 412/831] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:47:37.238594 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:37.243709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db14a57f-9011-4bd4-9f6e-da73aa15c6d8 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:37.243996 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db14a57f-9011-4bd4-9f6e-da73aa15c6d8 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:37.244271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db14a57f-9011-4bd4-9f6e-da73aa15c6d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:37.246985 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 420/832] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:37.251485 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30769c0-9511-42cb-8b45-f0e6add18770 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:37.267373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-db14a57f-9011-4bd4-9f6e-da73aa15c6d8 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:37.269374 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-db14a57f-9011-4bd4-9f6e-da73aa15c6d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:37.357731 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 413/833] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:37 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:47:37.388409 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30769c0-9511-42cb-8b45-f0e6add18770 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:37.426171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2d6ed6-5c02-49aa-a1fa-41139d6a3455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:37.426785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d6ed6-5c02-49aa-a1fa-41139d6a3455 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:37.427117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d6ed6-5c02-49aa-a1fa-41139d6a3455 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:37.427395 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2d6ed6-5c02-49aa-a1fa-41139d6a3455 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:37.428131 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2d6ed6-5c02-49aa-a1fa-41139d6a3455 None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=57c88f16c7df40adb917f49804fda21f) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:37.428439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2d6ed6-5c02-49aa-a1fa-41139d6a3455 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:37.429894 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2d6ed6-5c02-49aa-a1fa-41139d6a3455 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:37.440177 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 421/834] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:37 2022] GET /identity/v3/role_assignments?user.id=57c88f16c7df40adb917f49804fda21f&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=19c006412bec45dc8febd7cc86b9678f => generated 608 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:37.656340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db14a57f-9011-4bd4-9f6e-da73aa15c6d8 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:37.693057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fdbf80-f590-4b6c-96e3-46782df4b1d5 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:37.693402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fdbf80-f590-4b6c-96e3-46782df4b1d5 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:37.693630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fdbf80-f590-4b6c-96e3-46782df4b1d5 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:37.694957 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 422/835] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:47:37.699902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757dda81-cf01-4fa4-90fa-fb73e1d3b8ea None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:37.700183 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757dda81-cf01-4fa4-90fa-fb73e1d3b8ea None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:37.700458 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757dda81-cf01-4fa4-90fa-fb73e1d3b8ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:37.722923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-757dda81-cf01-4fa4-90fa-fb73e1d3b8ea None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:37.724152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-757dda81-cf01-4fa4-90fa-fb73e1d3b8ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:37.732044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:37.769148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db14a57f-9011-4bd4-9f6e-da73aa15c6d8 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:37.805173 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:37.811490 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf24a2d-c0ab-4b3b-85d8-d9c05de23118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:37.812137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf24a2d-c0ab-4b3b-85d8-d9c05de23118 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:37.812420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf24a2d-c0ab-4b3b-85d8-d9c05de23118 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:37.812696 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf24a2d-c0ab-4b3b-85d8-d9c05de23118 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:37.824020 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf24a2d-c0ab-4b3b-85d8-d9c05de23118 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:37.824318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 423/836] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:37 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:47:37.830775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-edf24a2d-c0ab-4b3b-85d8-d9c05de23118 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:47:37.830775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:37.830775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:37.830775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:37.830775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:37.830775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:37.830775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:37.830775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:37.830775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:37.830775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:37.830775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:37.830775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:37.830775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:37.830775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:37.830775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:37.830775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:37.831680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:37.831680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:37.831680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:37.831680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:37.831680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:37.831680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:37.831680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:37.831680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:37.831680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:37.831680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:37.831680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:37.831680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:37.831680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:37.831680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:37.831680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:37.831680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:37.832512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:37.832512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:37.832512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:37.832512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:37.832512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:37.832512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:37.832512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:37.832512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:37.832512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:37.832512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:37.832512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:47:37.832512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:37.832512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 414/837] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:47:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:37.838015 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757dda81-cf01-4fa4-90fa-fb73e1d3b8ea None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:37.944812 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757dda81-cf01-4fa4-90fa-fb73e1d3b8ea None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:37.979976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49dacea-5257-4c44-a083-e3b93c913346 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:37.980586 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49dacea-5257-4c44-a083-e3b93c913346 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:37.980870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49dacea-5257-4c44-a083-e3b93c913346 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:37.981176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49dacea-5257-4c44-a083-e3b93c913346 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:37.981848 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49dacea-5257-4c44-a083-e3b93c913346 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:37.982150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49dacea-5257-4c44-a083-e3b93c913346 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:37.983600 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49dacea-5257-4c44-a083-e3b93c913346 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:37.993064 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 424/838] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:47:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:37.998890 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf24a2d-c0ab-4b3b-85d8-d9c05de23118 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:38.102421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49dacea-5257-4c44-a083-e3b93c913346 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:38.104064 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf24a2d-c0ab-4b3b-85d8-d9c05de23118 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:38.138912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:38.153909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35e7a19-24cf-4393-a381-495b5f9bdbed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:38.155667 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35e7a19-24cf-4393-a381-495b5f9bdbed None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:38.155667 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35e7a19-24cf-4393-a381-495b5f9bdbed None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:38.156018 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35e7a19-24cf-4393-a381-495b5f9bdbed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:38.164555 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:38.171683 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35e7a19-24cf-4393-a381-495b5f9bdbed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:38.174063 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35e7a19-24cf-4393-a381-495b5f9bdbed None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:38.193415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-d35e7a19-24cf-4393-a381-495b5f9bdbed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:38.193415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:38.193415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:38.193415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:38.193415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:38.193415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:38.193415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:38.193415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:38.193415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:38.193415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:38.193415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:38.193415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:38.193415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:38.193415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:38.193415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:38.193415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:38.194461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:38.194461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:38.194461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:38.194461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:38.194461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:38.194461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:38.194461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:38.194461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:38.194461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:38.194461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:38.194461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:38.194461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:38.194461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:38.194461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:38.194461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:38.194461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:38.194461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:38.195407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:38.195407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:38.195407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:38.195407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 415/839] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:38.201346 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35e7a19-24cf-4393-a381-495b5f9bdbed None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:38.220784 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49dacea-5257-4c44-a083-e3b93c913346 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:38.259553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ddb148-69c0-4bf8-b605-52fe9a03b291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:38.260562 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ddb148-69c0-4bf8-b605-52fe9a03b291 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:38.261167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ddb148-69c0-4bf8-b605-52fe9a03b291 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:38.261695 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ddb148-69c0-4bf8-b605-52fe9a03b291 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:38.273707 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ddb148-69c0-4bf8-b605-52fe9a03b291 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:38.279706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-a3ddb148-69c0-4bf8-b605-52fe9a03b291 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:38.279706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:38.279706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:38.279706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:38.279706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:38.279706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:38.279706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:38.279706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:38.279706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:38.279706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:38.279706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:38.279706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:38.279706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:38.279706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:38.279706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:38.279706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:38.280612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:38.280612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:38.280612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:38.280612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:38.280612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:38.280612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:38.280612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:38.280612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:38.280612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:38.280612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:38.280612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:38.280612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:38.280612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:38.280612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:38.280612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:38.280612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:38.281472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:38.281472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:38.281472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:38.281472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:38.281472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:38.281472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:38.281472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:38.281472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:38.281472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:38.281472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:38.281472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:47:38.281472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:38.284547 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 425/840] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:47:38 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:38.290303 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ddb148-69c0-4bf8-b605-52fe9a03b291 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:38.325335 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35e7a19-24cf-4393-a381-495b5f9bdbed None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:38.363123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945f1748-011f-4c47-b657-d97c4692eaed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:38.364248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f1748-011f-4c47-b657-d97c4692eaed None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:38.364733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f1748-011f-4c47-b657-d97c4692eaed None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:38.365274 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f1748-011f-4c47-b657-d97c4692eaed None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:38.366146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945f1748-011f-4c47-b657-d97c4692eaed None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:38.366654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945f1748-011f-4c47-b657-d97c4692eaed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:38.368482 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945f1748-011f-4c47-b657-d97c4692eaed None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:38.387084 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 416/841] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:38.388388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945f1748-011f-4c47-b657-d97c4692eaed None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:38.418367 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ddb148-69c0-4bf8-b605-52fe9a03b291 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:38.436277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:38.441597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:38.456029 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bf7596-3539-4b7c-a515-f0a692ed82ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:38.456841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bf7596-3539-4b7c-a515-f0a692ed82ce None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:38.457305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bf7596-3539-4b7c-a515-f0a692ed82ce None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:38.457906 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bf7596-3539-4b7c-a515-f0a692ed82ce None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:38.459355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bf7596-3539-4b7c-a515-f0a692ed82ce None admin] RBAC: Adding query filter params (name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:38.460074 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bf7596-3539-4b7c-a515-f0a692ed82ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:38.461656 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bf7596-3539-4b7c-a515-f0a692ed82ce None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:38.472556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 426/842] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:47:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:38.474391 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da218db7-6ff2-49d9-b969-e2ae6ce92701 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:38.475644 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da218db7-6ff2-49d9-b969-e2ae6ce92701 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:38.475644 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da218db7-6ff2-49d9-b969-e2ae6ce92701 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:38.480452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 427/843] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:38.480452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da218db7-6ff2-49d9-b969-e2ae6ce92701 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:38.521646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945f1748-011f-4c47-b657-d97c4692eaed None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:38.557943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49724853-b8c1-4c19-a1f2-6cb7eaced920 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:38.558568 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49724853-b8c1-4c19-a1f2-6cb7eaced920 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:38.558845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49724853-b8c1-4c19-a1f2-6cb7eaced920 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:38.559112 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49724853-b8c1-4c19-a1f2-6cb7eaced920 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:38.566449 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49724853-b8c1-4c19-a1f2-6cb7eaced920 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:38.567941 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49724853-b8c1-4c19-a1f2-6cb7eaced920 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:38.580727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-49724853-b8c1-4c19-a1f2-6cb7eaced920 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 04 00:47:38.580727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:38.580727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:38.580727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:38.580727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:38.580727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:38.580727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:38.580727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:38.580727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:38.580727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:38.580727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:38.580727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:38.580727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:38.580727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:38.580727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:38.580727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:38.581981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:38.582912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:38.582912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:38.582912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:38.582912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:38.582912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:38.582912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:38.582912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:38.582912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:38.582912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 04 00:47:38.582912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:38.582912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 417/844] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:47:38 2022] GET /identity/v3/users/swift => generated 83 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:38.583504 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7301edf-4be1-4aa3-a300-a7942af997ca None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:38.583504 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7301edf-4be1-4aa3-a300-a7942af997ca None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:38.584067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7301edf-4be1-4aa3-a300-a7942af997ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:38.597187 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da218db7-6ff2-49d9-b969-e2ae6ce92701 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:38.603729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-f7301edf-4be1-4aa3-a300-a7942af997ca None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:38.605957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f7301edf-4be1-4aa3-a300-a7942af997ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:38.635733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cf1785-f8c6-4d4d-8a31-02a824fc0659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:38.636370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cf1785-f8c6-4d4d-8a31-02a824fc0659 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:38.636653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cf1785-f8c6-4d4d-8a31-02a824fc0659 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:38.636929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93cf1785-f8c6-4d4d-8a31-02a824fc0659 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:38.648339 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cf1785-f8c6-4d4d-8a31-02a824fc0659 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:38.649812 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93cf1785-f8c6-4d4d-8a31-02a824fc0659 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:38.663572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-93cf1785-f8c6-4d4d-8a31-02a824fc0659 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:38.663572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:38.663572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:38.663572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:38.663572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:38.663572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:38.663572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:38.663572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:38.663572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:38.663572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:38.663572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:38.663572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:38.663572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:38.663572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:38.663572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:38.663572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:38.664506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:38.664506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:38.664506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:38.664506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:38.664506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:38.664506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:38.664506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:38.664506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:38.664506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:38.664506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:38.664506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:38.664506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:38.664506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:38.664506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:38.664506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:38.664506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:38.664506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:38.665714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:38.665714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:38.665714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:38.665714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 428/845] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:38.665714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cf1785-f8c6-4d4d-8a31-02a824fc0659 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:38.705108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 418/846] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:38 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:38.705108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7301edf-4be1-4aa3-a300-a7942af997ca None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:38.705940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:38.740424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:38.780074 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93cf1785-f8c6-4d4d-8a31-02a824fc0659 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:38.820655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60c8231-3b84-48eb-b319-ee25d5820bd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:38.821491 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60c8231-3b84-48eb-b319-ee25d5820bd6 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:38.821801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60c8231-3b84-48eb-b319-ee25d5820bd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:38.822138 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60c8231-3b84-48eb-b319-ee25d5820bd6 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:38.822917 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60c8231-3b84-48eb-b319-ee25d5820bd6 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:38.823223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60c8231-3b84-48eb-b319-ee25d5820bd6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:38.824836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d60c8231-3b84-48eb-b319-ee25d5820bd6 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:38.828465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7301edf-4be1-4aa3-a300-a7942af997ca None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:38.844205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 429/847] 10.222.0.42 () {56 vars in 1135 bytes} [Sun Sep 4 00:47:38 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:38.850152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60c8231-3b84-48eb-b319-ee25d5820bd6 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:38.872460 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52b4dee-b2e3-454c-a962-2aaad57605aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:38.873388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52b4dee-b2e3-454c-a962-2aaad57605aa None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:38.873672 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52b4dee-b2e3-454c-a962-2aaad57605aa None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:38.873944 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52b4dee-b2e3-454c-a962-2aaad57605aa None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:38.874650 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52b4dee-b2e3-454c-a962-2aaad57605aa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:38.875485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52b4dee-b2e3-454c-a962-2aaad57605aa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:38.877745 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52b4dee-b2e3-454c-a962-2aaad57605aa None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:38.897167 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 419/848] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:38.899383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52b4dee-b2e3-454c-a962-2aaad57605aa None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:38.951497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:38.969016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d60c8231-3b84-48eb-b319-ee25d5820bd6 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:39.012152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:39.013419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac38f7e-4537-43d0-84aa-9ece381c7171 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:39.014155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac38f7e-4537-43d0-84aa-9ece381c7171 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:39.014294 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac38f7e-4537-43d0-84aa-9ece381c7171 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:39.014534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac38f7e-4537-43d0-84aa-9ece381c7171 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:39.022878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52b4dee-b2e3-454c-a962-2aaad57605aa None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:39.065721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292178f9-d06b-4856-922c-a54c99a5f16e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:39.066396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292178f9-d06b-4856-922c-a54c99a5f16e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:39.066905 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292178f9-d06b-4856-922c-a54c99a5f16e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:39.067015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292178f9-d06b-4856-922c-a54c99a5f16e None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:39.076668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292178f9-d06b-4856-922c-a54c99a5f16e None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:39.078153 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292178f9-d06b-4856-922c-a54c99a5f16e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:39.090321 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac38f7e-4537-43d0-84aa-9ece381c7171 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:39.091761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac38f7e-4537-43d0-84aa-9ece381c7171 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:39.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-292178f9-d06b-4856-922c-a54c99a5f16e None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 04 00:47:39.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:39.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:39.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:39.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:39.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:39.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:39.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:39.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:39.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:39.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:39.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:39.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:39.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:39.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:39.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:39.093159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:39.094121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:39.094121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:39.094121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:39.094121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:39.094121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:39.094121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:39.094121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:39.094121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:39.094121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 04 00:47:39.094121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:39.094121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 420/849] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 4 00:47:38 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:39.095341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292178f9-d06b-4856-922c-a54c99a5f16e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:39.188163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:39.202775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292178f9-d06b-4856-922c-a54c99a5f16e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:39.208662 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 430/850] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:38 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 04 00:47:39.208662 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac38f7e-4537-43d0-84aa-9ece381c7171 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:39.242628 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6499aacf-9561-45a3-b05c-024d80df384c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:39.243262 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6499aacf-9561-45a3-b05c-024d80df384c None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:39.243539 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6499aacf-9561-45a3-b05c-024d80df384c None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:39.243811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6499aacf-9561-45a3-b05c-024d80df384c None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:39.244648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6499aacf-9561-45a3-b05c-024d80df384c None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:39.246902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6499aacf-9561-45a3-b05c-024d80df384c None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:39.258500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-6499aacf-9561-45a3-b05c-024d80df384c None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 04 00:47:39.258500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:39.258500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:39.258500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:39.258500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:39.258500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:39.258500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:39.258500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:39.258500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 04 00:47:39.258500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:39.258500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:39.258500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:39.258500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:39.258500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:39.258500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:39.258500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:39.258500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:39.259553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:39.259553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:39.259553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:39.259553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:39.259553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:39.259553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:39.259553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:39.259553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:39.259553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:39.259553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:39.259553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:39.259553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:39.259553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:39.259553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:39.259553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:39.259553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:39.260403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:39.260403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:39.260403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:39.260403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:39.260403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:39.260403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:39.260403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:39.260403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:39.260403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:39.260403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:39.260403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:39.260403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:39.260403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 04 00:47:39.260403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:39.260403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 421/851] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:47:39 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:39.261230 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6499aacf-9561-45a3-b05c-024d80df384c None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:39.278483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:39.316101 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac38f7e-4537-43d0-84aa-9ece381c7171 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:39.352409 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff10855-0a50-4034-887b-a1f14591ecb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:39.353012 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff10855-0a50-4034-887b-a1f14591ecb8 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:39.353312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff10855-0a50-4034-887b-a1f14591ecb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:39.353579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff10855-0a50-4034-887b-a1f14591ecb8 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:39.354303 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff10855-0a50-4034-887b-a1f14591ecb8 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:39.354598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff10855-0a50-4034-887b-a1f14591ecb8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:39.356015 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff10855-0a50-4034-887b-a1f14591ecb8 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:39.371328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6499aacf-9561-45a3-b05c-024d80df384c None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:39.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 431/852] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Sep 4 00:47:39 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:39.379179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff10855-0a50-4034-887b-a1f14591ecb8 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:39.407832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414be0a6-20a7-4414-bc33-6f9b170a0d65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:39.409132 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414be0a6-20a7-4414-bc33-6f9b170a0d65 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:39.409407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414be0a6-20a7-4414-bc33-6f9b170a0d65 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:39.409676 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414be0a6-20a7-4414-bc33-6f9b170a0d65 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:39.420104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414be0a6-20a7-4414-bc33-6f9b170a0d65 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:39.421582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414be0a6-20a7-4414-bc33-6f9b170a0d65 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:39.426021 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:39.440081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-414be0a6-20a7-4414-bc33-6f9b170a0d65 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:39.440081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:39.440081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:39.440081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:39.440081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:39.440081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:39.440081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:39.440081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:39.440081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:39.440081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:39.440081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:39.440081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:39.440081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:39.440081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:39.440081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:39.440081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:39.441100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:39.441100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:39.441100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:39.441100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:39.441100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:39.441100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:39.441100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:39.441100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:39.441100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:39.441100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:39.441100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:39.441100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:39.441100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:39.441100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:39.441100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:39.441100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:39.441100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:39.441981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:39.441981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:39.441981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:39.443811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 422/853] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:39.444807 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414be0a6-20a7-4414-bc33-6f9b170a0d65 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:39.489798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff10855-0a50-4034-887b-a1f14591ecb8 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:39.516586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:39.523441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5d8633-6ad8-43a7-9e3c-34a2e09f416b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:39.524046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5d8633-6ad8-43a7-9e3c-34a2e09f416b None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:39.524322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5d8633-6ad8-43a7-9e3c-34a2e09f416b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:39.524592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5d8633-6ad8-43a7-9e3c-34a2e09f416b None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:39.525245 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5d8633-6ad8-43a7-9e3c-34a2e09f416b None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:39.525540 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5d8633-6ad8-43a7-9e3c-34a2e09f416b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:39.526905 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5d8633-6ad8-43a7-9e3c-34a2e09f416b None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:39.542308 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 432/854] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:47:39 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:39.542308 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5d8633-6ad8-43a7-9e3c-34a2e09f416b None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:39.551826 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414be0a6-20a7-4414-bc33-6f9b170a0d65 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:39.585463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cd0673-c4bd-44f8-933f-25712c0ac923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:39.586063 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cd0673-c4bd-44f8-933f-25712c0ac923 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:39.586345 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cd0673-c4bd-44f8-933f-25712c0ac923 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:39.586637 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cd0673-c4bd-44f8-933f-25712c0ac923 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:39.632882 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5d8633-6ad8-43a7-9e3c-34a2e09f416b None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:39.643329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:39.654159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cd0673-c4bd-44f8-933f-25712c0ac923 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:39.655591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cd0673-c4bd-44f8-933f-25712c0ac923 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:39.666602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555a8551-be80-483b-aa7e-55def6568009 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:39.667203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555a8551-be80-483b-aa7e-55def6568009 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:39.667480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555a8551-be80-483b-aa7e-55def6568009 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:39.667751 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555a8551-be80-483b-aa7e-55def6568009 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:39.668393 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555a8551-be80-483b-aa7e-55def6568009 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:39.668691 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555a8551-be80-483b-aa7e-55def6568009 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:39.670069 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555a8551-be80-483b-aa7e-55def6568009 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:39.682512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 433/855] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:39.683679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27acf5f6-5ece-425f-8eb7-fb2ce731a336 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:39.683971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27acf5f6-5ece-425f-8eb7-fb2ce731a336 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:39.684244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27acf5f6-5ece-425f-8eb7-fb2ce731a336 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:39.685457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 434/856] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:39.686146 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27acf5f6-5ece-425f-8eb7-fb2ce731a336 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:39.724665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:39.759625 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 423/857] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:39 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 04 00:47:39.760452 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cd0673-c4bd-44f8-933f-25712c0ac923 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:39.790412 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27acf5f6-5ece-425f-8eb7-fb2ce731a336 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:39.822472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfd7767-7e7d-4884-9495-2576203b6967 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:39.823065 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfd7767-7e7d-4884-9495-2576203b6967 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:39.823348 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfd7767-7e7d-4884-9495-2576203b6967 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:39.823617 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfd7767-7e7d-4884-9495-2576203b6967 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:39.824235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edfd7767-7e7d-4884-9495-2576203b6967 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:39.824535 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edfd7767-7e7d-4884-9495-2576203b6967 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:39.825925 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edfd7767-7e7d-4884-9495-2576203b6967 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:39.830762 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 435/858] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:47:39 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:39.831897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaf238c-b44b-443b-83d2-4a39f8e16e80 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:39.832173 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaf238c-b44b-443b-83d2-4a39f8e16e80 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:39.832439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaf238c-b44b-443b-83d2-4a39f8e16e80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:39.859743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-faaf238c-b44b-443b-83d2-4a39f8e16e80 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:39.861979 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-faaf238c-b44b-443b-83d2-4a39f8e16e80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:39.866475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cd0673-c4bd-44f8-933f-25712c0ac923 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:39.895085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:39.904175 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d12b55-1b0e-4a14-ab7f-507bc8b5ab38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:39.904816 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d12b55-1b0e-4a14-ab7f-507bc8b5ab38 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:39.908546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d12b55-1b0e-4a14-ab7f-507bc8b5ab38 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:39.908546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d12b55-1b0e-4a14-ab7f-507bc8b5ab38 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:39.913652 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d12b55-1b0e-4a14-ab7f-507bc8b5ab38 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:39.915165 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d12b55-1b0e-4a14-ab7f-507bc8b5ab38 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:39.927607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-85d12b55-1b0e-4a14-ab7f-507bc8b5ab38 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:39.927607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:39.927607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:39.927607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:39.927607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:39.927607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:39.927607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:39.927607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:39.927607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:39.927607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:39.927607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:39.927607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:39.927607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:39.927607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:39.927607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:39.928765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:39.928765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:39.928765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:39.928765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:39.928765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:39.928765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:39.928765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:39.928765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:39.928765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:39.928765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:39.928765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:39.928765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:39.928765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:39.928765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:39.928765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:39.928765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:39.928765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:39.929729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:39.929729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:39.929729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:39.929729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:39.929729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:39.929729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:39.929729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:39.929729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:39.929729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 424/859] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:39.930218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d12b55-1b0e-4a14-ab7f-507bc8b5ab38 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:39.948868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 436/860] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:39 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:39.949792 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faaf238c-b44b-443b-83d2-4a39f8e16e80 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:39.965786 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:40.054676 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d12b55-1b0e-4a14-ab7f-507bc8b5ab38 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:40.077813 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faaf238c-b44b-443b-83d2-4a39f8e16e80 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:40.101860 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8602f057-be69-40e2-a599-b9656dd7ed7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:40.103178 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8602f057-be69-40e2-a599-b9656dd7ed7e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:40.103540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8602f057-be69-40e2-a599-b9656dd7ed7e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:40.103892 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8602f057-be69-40e2-a599-b9656dd7ed7e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:40.112985 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f52a16a-25aa-49d0-bf0a-7900f54a02ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:40.113621 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f52a16a-25aa-49d0-bf0a-7900f54a02ed None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:40.113898 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f52a16a-25aa-49d0-bf0a-7900f54a02ed None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:40.114171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f52a16a-25aa-49d0-bf0a-7900f54a02ed None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:40.114865 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f52a16a-25aa-49d0-bf0a-7900f54a02ed None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:40.115167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f52a16a-25aa-49d0-bf0a-7900f54a02ed None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:40.116675 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f52a16a-25aa-49d0-bf0a-7900f54a02ed None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:40.117022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8602f057-be69-40e2-a599-b9656dd7ed7e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:40.118725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8602f057-be69-40e2-a599-b9656dd7ed7e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:40.129664 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 437/861] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:40.135141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f52a16a-25aa-49d0-bf0a-7900f54a02ed None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:40.142190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-8602f057-be69-40e2-a599-b9656dd7ed7e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:40.142190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:40.142190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:40.142190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:40.142190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:40.142190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:40.142190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:40.142190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:40.142190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:40.142190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:40.142190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:40.142190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:40.142190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:40.142190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:40.142190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:40.142190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:40.144232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:40.144232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:40.144232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:40.144232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:40.144232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:40.144232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:40.144232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:40.144232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:40.144232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:40.144232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:40.144232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:40.144232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:40.144232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:40.144232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:40.144232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:40.144232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:40.144232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:40.145349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:40.145349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:40.145349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:40.146527 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 425/862] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:40.151614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8602f057-be69-40e2-a599-b9656dd7ed7e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:40.169619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:40.240708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:40.262336 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8602f057-be69-40e2-a599-b9656dd7ed7e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:40.268093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f52a16a-25aa-49d0-bf0a-7900f54a02ed None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:40.301116 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b702e6a1-5668-48d2-9c1b-21106d7f5828 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:40.302089 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b702e6a1-5668-48d2-9c1b-21106d7f5828 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:40.302580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b702e6a1-5668-48d2-9c1b-21106d7f5828 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:40.303058 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b702e6a1-5668-48d2-9c1b-21106d7f5828 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:40.303653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe13f1d-3c08-4560-a160-f23937f1a677 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:40.304165 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b702e6a1-5668-48d2-9c1b-21106d7f5828 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:40.304286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe13f1d-3c08-4560-a160-f23937f1a677 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:40.304520 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe13f1d-3c08-4560-a160-f23937f1a677 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:40.304793 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe13f1d-3c08-4560-a160-f23937f1a677 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:40.305209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b702e6a1-5668-48d2-9c1b-21106d7f5828 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:40.305552 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe13f1d-3c08-4560-a160-f23937f1a677 None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=123ab6ff796c4e599ce1d6f4095bd898) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:40.305861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe13f1d-3c08-4560-a160-f23937f1a677 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:40.307228 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b702e6a1-5668-48d2-9c1b-21106d7f5828 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:40.307336 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe13f1d-3c08-4560-a160-f23937f1a677 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:40.313117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 438/863] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:40 2022] GET /identity/v3/role_assignments?user.id=123ab6ff796c4e599ce1d6f4095bd898&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=19c006412bec45dc8febd7cc86b9678f => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:40.321206 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe13f1d-3c08-4560-a160-f23937f1a677 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:40.323721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 426/864] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:40.329853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b702e6a1-5668-48d2-9c1b-21106d7f5828 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:40.441302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:40.457861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe13f1d-3c08-4560-a160-f23937f1a677 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:40.461882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b702e6a1-5668-48d2-9c1b-21106d7f5828 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:40.510984 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1851d9-52da-49e3-beab-770eb1eec281 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:40.512452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1851d9-52da-49e3-beab-770eb1eec281 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:40.513397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1851d9-52da-49e3-beab-770eb1eec281 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:40.513669 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1851d9-52da-49e3-beab-770eb1eec281 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:40.514635 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1851d9-52da-49e3-beab-770eb1eec281 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:40.516597 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1851d9-52da-49e3-beab-770eb1eec281 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:40.518138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73dc7fed-4e7b-4f15-9a10-061567cf0daa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:40.518852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dc7fed-4e7b-4f15-9a10-061567cf0daa None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:40.519129 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dc7fed-4e7b-4f15-9a10-061567cf0daa None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:40.519400 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dc7fed-4e7b-4f15-9a10-061567cf0daa None admin] PATH_INFO: `/v3/projects/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:40.527669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73dc7fed-4e7b-4f15-9a10-061567cf0daa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:40.528900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-3c1851d9-52da-49e3-beab-770eb1eec281 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 04 00:47:40.528900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:40.528900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:40.528900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:40.528900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:40.528900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:40.528900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:40.528900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:40.528900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 04 00:47:40.528900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:40.528900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:40.528900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:40.528900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:40.528900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:40.528900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:40.528900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:40.529819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:40.529819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:40.529819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:40.529819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:40.529819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:40.529819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:40.529819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:40.529819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:40.529819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:40.529819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:40.529819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:40.529819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:40.529819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:40.529819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:40.529819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:40.529819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:40.529819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:40.530708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:40.530708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:40.530708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:40.530708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:40.530708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:40.530708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:40.530708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:40.530708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:40.530708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:40.530708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:40.530708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:40.530708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:40.530708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 04 00:47:40.530708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:40.531446 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73dc7fed-4e7b-4f15-9a10-061567cf0daa None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:40.532219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 439/865] 10.222.0.42 () {56 vars in 1132 bytes} [Sun Sep 4 00:47:40 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:40.533768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:40.537869 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1851d9-52da-49e3-beab-770eb1eec281 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:40.545359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-73dc7fed-4e7b-4f15-9a10-061567cf0daa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:40.545359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:40.545359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:40.545359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:40.545359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:40.545359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:40.545359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:40.545359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:40.545359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:40.545359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:40.545359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:40.545359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:40.545359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:40.545359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:40.545359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:40.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:40.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:40.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:40.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:40.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:40.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:40.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:40.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:40.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:40.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:40.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:40.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:40.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:40.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:40.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:40.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:40.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:40.551127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:40.551127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:40.551127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:40.551127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:40.551127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:40.551127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:40.551127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:40.551127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:40.551127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 427/866] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:40.553417 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73dc7fed-4e7b-4f15-9a10-061567cf0daa None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:40.651195 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1851d9-52da-49e3-beab-770eb1eec281 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:40.664787 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73dc7fed-4e7b-4f15-9a10-061567cf0daa None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:40.687618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db850d19-48fe-4764-bb8e-6a185bf00385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:40.688297 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db850d19-48fe-4764-bb8e-6a185bf00385 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:40.688501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db850d19-48fe-4764-bb8e-6a185bf00385 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:40.688751 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db850d19-48fe-4764-bb8e-6a185bf00385 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:40.690065 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db850d19-48fe-4764-bb8e-6a185bf00385 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:40.690367 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db850d19-48fe-4764-bb8e-6a185bf00385 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:40.691863 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db850d19-48fe-4764-bb8e-6a185bf00385 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:40.701529 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 440/867] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 4 00:47:40 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:40.702568 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db850d19-48fe-4764-bb8e-6a185bf00385 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:40.704236 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b45d7cc-4f34-4dd9-883b-7de09f7cdeee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:40.705376 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b45d7cc-4f34-4dd9-883b-7de09f7cdeee None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:40.705887 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b45d7cc-4f34-4dd9-883b-7de09f7cdeee None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:40.706383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b45d7cc-4f34-4dd9-883b-7de09f7cdeee None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:40.707326 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b45d7cc-4f34-4dd9-883b-7de09f7cdeee None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:40.707836 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b45d7cc-4f34-4dd9-883b-7de09f7cdeee None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:40.709481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b45d7cc-4f34-4dd9-883b-7de09f7cdeee None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:40.722580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 428/868] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:40.727331 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:40.729363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b45d7cc-4f34-4dd9-883b-7de09f7cdeee None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:40.800379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:40.827012 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db850d19-48fe-4764-bb8e-6a185bf00385 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:40.845228 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b45d7cc-4f34-4dd9-883b-7de09f7cdeee None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:40.867946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5146ec9e-6f7b-42f8-8b8d-9d674528d111 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:40.868935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5146ec9e-6f7b-42f8-8b8d-9d674528d111 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:40.869483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5146ec9e-6f7b-42f8-8b8d-9d674528d111 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:40.870063 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5146ec9e-6f7b-42f8-8b8d-9d674528d111 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:40.871004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5146ec9e-6f7b-42f8-8b8d-9d674528d111 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:40.871530 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5146ec9e-6f7b-42f8-8b8d-9d674528d111 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:40.873613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5146ec9e-6f7b-42f8-8b8d-9d674528d111 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:40.879602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 441/869] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 4 00:47:40 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:40.894010 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a54bf7-8907-4316-b621-21662f00cf87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:40.899560 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a54bf7-8907-4316-b621-21662f00cf87 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:40.899560 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a54bf7-8907-4316-b621-21662f00cf87 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:40.899560 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a54bf7-8907-4316-b621-21662f00cf87 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:40.899560 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a54bf7-8907-4316-b621-21662f00cf87 None admin] RBAC: Adding query filter params (role.id=df912a77fbaa479db172f76a214f8bc8, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=70cef7c271e046a3b49230057cc2d28f) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:40.899560 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a54bf7-8907-4316-b621-21662f00cf87 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:40.899560 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a54bf7-8907-4316-b621-21662f00cf87 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:40.904924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 429/870] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:40 2022] GET /identity/v3/role_assignments?user.id=70cef7c271e046a3b49230057cc2d28f&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=df912a77fbaa479db172f76a214f8bc8 => generated 608 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:41.845435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f674a5-607d-419c-ba54-3f65be98f977 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:41.846212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f674a5-607d-419c-ba54-3f65be98f977 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:41.846212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f674a5-607d-419c-ba54-3f65be98f977 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:41.847446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 442/871] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:41.852923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870f71b1-95b1-4880-807b-d748cb813329 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:41.853756 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870f71b1-95b1-4880-807b-d748cb813329 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:41.854248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870f71b1-95b1-4880-807b-d748cb813329 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:41.872921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-870f71b1-95b1-4880-807b-d748cb813329 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:41.874784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-870f71b1-95b1-4880-807b-d748cb813329 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:41.947100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 430/872] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:41 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:42.216282 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f674a5-607d-419c-ba54-3f65be98f977 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:42.302302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:42.322620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f674a5-607d-419c-ba54-3f65be98f977 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:42.362476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76beefbc-a233-429f-8dbd-38e01834f620 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:42.363136 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76beefbc-a233-429f-8dbd-38e01834f620 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:42.363428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76beefbc-a233-429f-8dbd-38e01834f620 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:42.363679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76beefbc-a233-429f-8dbd-38e01834f620 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:42.364341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76beefbc-a233-429f-8dbd-38e01834f620 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:42.364795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cea2e2-9626-4ae9-a9e7-7802c8f3b047 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:42.365106 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cea2e2-9626-4ae9-a9e7-7802c8f3b047 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:42.365382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cea2e2-9626-4ae9-a9e7-7802c8f3b047 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:42.365748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76beefbc-a233-429f-8dbd-38e01834f620 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:42.368096 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 431/873] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:42.372310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d72e38-2fe6-460a-90a5-c84f77a3179d None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:42.372587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d72e38-2fe6-460a-90a5-c84f77a3179d None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:42.372853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d72e38-2fe6-460a-90a5-c84f77a3179d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:42.391819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-e0d72e38-2fe6-460a-90a5-c84f77a3179d None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:42.393591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e0d72e38-2fe6-460a-90a5-c84f77a3179d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:42.443109 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 443/874] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 4 00:47:42 2022] POST /identity/v3/services => generated 251 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:47:42.453131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:42.484131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 432/875] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:42 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:47:42.753000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76beefbc-a233-429f-8dbd-38e01834f620 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:42.856428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76beefbc-a233-429f-8dbd-38e01834f620 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:42.867002 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe31e80-5014-4f8b-beb1-32e619132660 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:42.867455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe31e80-5014-4f8b-beb1-32e619132660 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:42.867556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe31e80-5014-4f8b-beb1-32e619132660 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:42.869138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 433/876] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:42.874407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326b2d3f-e768-476c-b4bf-d16be9381a54 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:42.874796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326b2d3f-e768-476c-b4bf-d16be9381a54 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:42.875079 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326b2d3f-e768-476c-b4bf-d16be9381a54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:42.893046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f16d44-4130-4747-8140-1ee3f149705d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:42.893704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f16d44-4130-4747-8140-1ee3f149705d None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:42.893982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f16d44-4130-4747-8140-1ee3f149705d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:42.894251 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f16d44-4130-4747-8140-1ee3f149705d None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:42.894484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-326b2d3f-e768-476c-b4bf-d16be9381a54 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:42.894830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f16d44-4130-4747-8140-1ee3f149705d None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:42.895684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-326b2d3f-e768-476c-b4bf-d16be9381a54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:42.896252 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f16d44-4130-4747-8140-1ee3f149705d None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:42.912318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-36f16d44-4130-4747-8140-1ee3f149705d None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 04 00:47:42.912318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:42.912318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:42.912318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:42.912318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:42.912318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:42.912318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:42.912318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:42.912318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 04 00:47:42.912318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:42.912318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:42.912318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:42.912318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:42.912318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:42.912318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:42.912318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:42.912318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:42.913397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:42.913397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:42.913397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:42.913397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:42.913397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:42.913397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:42.913397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:42.913397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:42.913397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:42.913397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:42.913397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:42.913397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:42.913397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:42.913397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:42.913397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:42.913397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:42.914223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:42.914223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:42.914223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:42.914223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:42.914223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:42.914223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:42.914223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:42.914223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:42.914223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:42.914223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:42.914223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:42.914223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:42.914223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 04 00:47:42.914223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:42.915697 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 444/877] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 4 00:47:42 2022] GET /identity/v3/services/object-store => generated 93 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:42.921528 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f16d44-4130-4747-8140-1ee3f149705d None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:42.926538 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:42.982416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 434/878] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:42 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:47:42.995087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2127c02d-3fed-47c3-bb75-97f8b3814721 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:42.995383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2127c02d-3fed-47c3-bb75-97f8b3814721 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:42.995665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2127c02d-3fed-47c3-bb75-97f8b3814721 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:42.997993 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 435/879] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:43.003485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cbd695-97e9-4d8a-9032-34983179a3a7 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:43.003719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cbd695-97e9-4d8a-9032-34983179a3a7 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:43.003999 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cbd695-97e9-4d8a-9032-34983179a3a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:43.022768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-46cbd695-97e9-4d8a-9032-34983179a3a7 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:43.023976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-46cbd695-97e9-4d8a-9032-34983179a3a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:43.058894 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f16d44-4130-4747-8140-1ee3f149705d None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:43.101493 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 436/880] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:43 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:43.107378 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc33367-eac6-4d3f-b52c-d978513bfcb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:43.108040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc33367-eac6-4d3f-b52c-d978513bfcb7 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:43.108322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc33367-eac6-4d3f-b52c-d978513bfcb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:43.108595 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc33367-eac6-4d3f-b52c-d978513bfcb7 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:43.109301 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc33367-eac6-4d3f-b52c-d978513bfcb7 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:43.109604 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc33367-eac6-4d3f-b52c-d978513bfcb7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:43.111072 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc33367-eac6-4d3f-b52c-d978513bfcb7 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:43.116292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 445/881] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 4 00:47:42 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:43.121884 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cbd695-97e9-4d8a-9032-34983179a3a7 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:43.168082 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:43.228924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cbd695-97e9-4d8a-9032-34983179a3a7 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:43.260244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc33367-eac6-4d3f-b52c-d978513bfcb7 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:43.265149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6a8404-7ae4-47bc-9fd5-11f15b2425f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:43.265992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6a8404-7ae4-47bc-9fd5-11f15b2425f2 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:43.266469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6a8404-7ae4-47bc-9fd5-11f15b2425f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:43.266961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6a8404-7ae4-47bc-9fd5-11f15b2425f2 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:43.268346 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6a8404-7ae4-47bc-9fd5-11f15b2425f2 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:43.268853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6a8404-7ae4-47bc-9fd5-11f15b2425f2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:43.270441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6a8404-7ae4-47bc-9fd5-11f15b2425f2 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:43.281209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 437/882] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 4 00:47:43 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:43.334198 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:43.373963 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc33367-eac6-4d3f-b52c-d978513bfcb7 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:43.374157 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6a8404-7ae4-47bc-9fd5-11f15b2425f2 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:43.417658 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae439887-7613-4660-a382-63a5b4acc2a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:43.418276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae439887-7613-4660-a382-63a5b4acc2a5 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:43.418554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae439887-7613-4660-a382-63a5b4acc2a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:43.418825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae439887-7613-4660-a382-63a5b4acc2a5 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:43.419482 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae439887-7613-4660-a382-63a5b4acc2a5 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:43.420974 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae439887-7613-4660-a382-63a5b4acc2a5 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:43.507463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6a8404-7ae4-47bc-9fd5-11f15b2425f2 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:43.509187 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:43.547754 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 446/883] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 4 00:47:43 2022] POST /identity/v3/services => generated 269 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 00:47:43.558925 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78427ba0-74c3-4cbf-8068-8d9a1aaac820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:43.559685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78427ba0-74c3-4cbf-8068-8d9a1aaac820 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:43.559968 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78427ba0-74c3-4cbf-8068-8d9a1aaac820 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:43.560242 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78427ba0-74c3-4cbf-8068-8d9a1aaac820 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:43.574515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78427ba0-74c3-4cbf-8068-8d9a1aaac820 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:43.580797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-78427ba0-74c3-4cbf-8068-8d9a1aaac820 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 04 00:47:43.580797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:43.580797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:43.580797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:43.580797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:43.580797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:43.580797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:43.580797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:43.580797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:43.580797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:43.580797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:43.580797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:43.580797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:43.580797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:43.580797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:43.580797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:43.581822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:43.581822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:43.581822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:43.581822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:43.581822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:43.581822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:43.581822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:43.581822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:43.581822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:43.581822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:43.581822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:43.581822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:43.581822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:43.581822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:43.581822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:43.581822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:43.583241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:43.583241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:43.583241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:43.583241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:43.583241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:43.583241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:43.583241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:43.583241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:43.583241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:43.583241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:43.583241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 04 00:47:43.583241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:43.583241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 438/884] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Sep 4 00:47:43 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:43.588111 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae439887-7613-4660-a382-63a5b4acc2a5 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:43.715677 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae439887-7613-4660-a382-63a5b4acc2a5 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:43.745661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:43.754544 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6f74d5-bf15-47d1-bc63-4f1833169539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:43.755165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6f74d5-bf15-47d1-bc63-4f1833169539 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:43.759193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6f74d5-bf15-47d1-bc63-4f1833169539 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:43.759193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6f74d5-bf15-47d1-bc63-4f1833169539 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:43.759193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6f74d5-bf15-47d1-bc63-4f1833169539 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:43.759193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6f74d5-bf15-47d1-bc63-4f1833169539 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:43.759193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a6f74d5-bf15-47d1-bc63-4f1833169539 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:43.767505 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 447/885] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Sep 4 00:47:43 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:43.773352 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78427ba0-74c3-4cbf-8068-8d9a1aaac820 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:43.879298 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78427ba0-74c3-4cbf-8068-8d9a1aaac820 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:43.914864 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a67cd8-2f3f-40c4-b159-4a58dfde9303 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:43.915500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a67cd8-2f3f-40c4-b159-4a58dfde9303 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:43.915783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a67cd8-2f3f-40c4-b159-4a58dfde9303 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:43.916058 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a67cd8-2f3f-40c4-b159-4a58dfde9303 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:43.928097 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a67cd8-2f3f-40c4-b159-4a58dfde9303 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:43.929582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a67cd8-2f3f-40c4-b159-4a58dfde9303 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:43.944510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-94a67cd8-2f3f-40c4-b159-4a58dfde9303 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:43.944510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:43.944510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:43.944510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:43.944510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:43.944510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:43.944510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:43.944510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:43.944510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:43.944510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:43.944510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:43.944510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:43.944510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:43.944510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:43.944510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:43.944510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:43.945524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:43.945524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:43.945524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:43.945524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:43.945524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:43.945524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:43.945524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:43.945524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:43.945524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:43.945524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:43.945524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:43.945524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:43.945524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:43.945524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:43.945524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:43.945524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:43.945524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:43.946478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:43.946478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:43.946478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:43.946478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 439/886] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:43.951707 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6f74d5-bf15-47d1-bc63-4f1833169539 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:44.056037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6f74d5-bf15-47d1-bc63-4f1833169539 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:44.095768 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745f9057-ad6f-4406-ab99-a52e0ad08b90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:44.096387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745f9057-ad6f-4406-ab99-a52e0ad08b90 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:44.096664 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745f9057-ad6f-4406-ab99-a52e0ad08b90 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:44.096935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745f9057-ad6f-4406-ab99-a52e0ad08b90 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:44.097633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745f9057-ad6f-4406-ab99-a52e0ad08b90 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:44.097934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745f9057-ad6f-4406-ab99-a52e0ad08b90 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:44.099299 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745f9057-ad6f-4406-ab99-a52e0ad08b90 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:44.111276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 448/887] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:44.116549 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a67cd8-2f3f-40c4-b159-4a58dfde9303 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:44.145528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:44.227787 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a67cd8-2f3f-40c4-b159-4a58dfde9303 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:44.264557 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93322243-3e99-4389-b03e-570303e364fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:44.265227 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93322243-3e99-4389-b03e-570303e364fa None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:44.265520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93322243-3e99-4389-b03e-570303e364fa None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:44.265794 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93322243-3e99-4389-b03e-570303e364fa None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:44.273406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93322243-3e99-4389-b03e-570303e364fa None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:44.274867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93322243-3e99-4389-b03e-570303e364fa None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:44.286680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-93322243-3e99-4389-b03e-570303e364fa None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 04 00:47:44.286680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:44.286680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:44.286680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:44.286680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:44.286680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:44.286680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:44.286680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:44.286680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:44.286680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:44.286680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:44.286680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:44.286680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:44.286680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:44.286680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:44.286680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:44.287623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:44.288550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:44.288550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:44.288550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:44.288550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:44.288550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:44.288550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:44.288550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:44.288550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:44.288550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 04 00:47:44.288550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:44.288550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 440/888] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 4 00:47:44 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:44.293792 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745f9057-ad6f-4406-ab99-a52e0ad08b90 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:44.346055 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:44.403182 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745f9057-ad6f-4406-ab99-a52e0ad08b90 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:44.438583 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cde745f-4909-48d0-852c-509e5b35094c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:44.440059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cde745f-4909-48d0-852c-509e5b35094c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:44.440484 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cde745f-4909-48d0-852c-509e5b35094c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:44.440924 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cde745f-4909-48d0-852c-509e5b35094c None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:44.441867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cde745f-4909-48d0-852c-509e5b35094c None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:44.442312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cde745f-4909-48d0-852c-509e5b35094c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:44.443868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cde745f-4909-48d0-852c-509e5b35094c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:44.462813 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 449/889] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Sep 4 00:47:44 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:44.471755 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93322243-3e99-4389-b03e-570303e364fa None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:44.525446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4a2f33-8521-4a45-8ad9-b16a44e4b1e4 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:44.525762 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4a2f33-8521-4a45-8ad9-b16a44e4b1e4 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:44.526000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4a2f33-8521-4a45-8ad9-b16a44e4b1e4 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:44.527633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 450/890] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:44.532754 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3647abbf-cbcd-4fba-80a3-344f8e9b6c97 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:44.533695 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3647abbf-cbcd-4fba-80a3-344f8e9b6c97 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:44.534154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3647abbf-cbcd-4fba-80a3-344f8e9b6c97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:44.550373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:44.554683 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-3647abbf-cbcd-4fba-80a3-344f8e9b6c97 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:44.555888 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-3647abbf-cbcd-4fba-80a3-344f8e9b6c97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:44.586375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93322243-3e99-4389-b03e-570303e364fa None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:44.625971 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e74b29-5ccb-4296-87d9-da88a8458fea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:44.626739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e74b29-5ccb-4296-87d9-da88a8458fea None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:44.627230 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e74b29-5ccb-4296-87d9-da88a8458fea None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:44.627684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e74b29-5ccb-4296-87d9-da88a8458fea None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:44.638333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 451/891] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:44 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:44.701981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e74b29-5ccb-4296-87d9-da88a8458fea None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:44.703806 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e74b29-5ccb-4296-87d9-da88a8458fea None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:44.776110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:44.810633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 441/892] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:44 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 04 00:47:44.817501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3647abbf-cbcd-4fba-80a3-344f8e9b6c97 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:44.859560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:44.928662 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3647abbf-cbcd-4fba-80a3-344f8e9b6c97 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:44.935564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e74b29-5ccb-4296-87d9-da88a8458fea None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:44.964529 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e70844-31ed-4bff-b2a8-70aa40f22e17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:44.965210 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e70844-31ed-4bff-b2a8-70aa40f22e17 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:44.965400 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e70844-31ed-4bff-b2a8-70aa40f22e17 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:44.965678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e70844-31ed-4bff-b2a8-70aa40f22e17 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:44.976220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e70844-31ed-4bff-b2a8-70aa40f22e17 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:44.977730 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37e70844-31ed-4bff-b2a8-70aa40f22e17 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:44.994364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-37e70844-31ed-4bff-b2a8-70aa40f22e17 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:44.994364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:44.994364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:44.994364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:44.994364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:44.994364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:44.994364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:44.994364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:44.994364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:44.994364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:44.994364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:44.994364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:44.994364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:44.994364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:44.994364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:44.994364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:44.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:44.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:44.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:44.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:44.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:44.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:44.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:44.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:44.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:44.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:44.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:44.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:44.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:44.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:44.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:44.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:44.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:44.996227 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:44.996227 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:44.996227 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:44.996227 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 452/893] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:45.003176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e70844-31ed-4bff-b2a8-70aa40f22e17 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:45.058388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e74b29-5ccb-4296-87d9-da88a8458fea None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:45.099970 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45858d80-778c-497a-b992-3927457a232f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:45.100606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45858d80-778c-497a-b992-3927457a232f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:45.100888 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45858d80-778c-497a-b992-3927457a232f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:45.101204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45858d80-778c-497a-b992-3927457a232f None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:45.101797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45858d80-778c-497a-b992-3927457a232f None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:45.103311 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45858d80-778c-497a-b992-3927457a232f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:45.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-45858d80-778c-497a-b992-3927457a232f None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 04 00:47:45.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:45.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:45.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:45.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:45.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:45.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:45.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:45.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 04 00:47:45.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:45.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:45.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:45.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:45.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:45.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:45.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:45.113569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:45.114575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:45.114575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:45.114575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:45.114575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:45.114575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:45.114575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:45.114575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:45.114575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:45.114575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:45.114575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:45.114575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:45.114575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:45.114575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:45.114575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:45.114575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:45.114575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:45.115387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:45.115387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:45.115387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:45.115387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:45.115387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:45.115387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:45.115387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:45.115387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:45.115387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:45.115387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:45.115387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:45.115387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:45.115387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 04 00:47:45.115387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:45.115387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 442/894] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:47:44 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:45.122235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45858d80-778c-497a-b992-3927457a232f None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:45.129541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:45.139679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e70844-31ed-4bff-b2a8-70aa40f22e17 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:45.159845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:45.177651 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cf059e-ee3c-4f69-897b-452bf3d9d031 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:45.178664 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cf059e-ee3c-4f69-897b-452bf3d9d031 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:45.179150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cf059e-ee3c-4f69-897b-452bf3d9d031 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:45.179628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cf059e-ee3c-4f69-897b-452bf3d9d031 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:45.180498 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8cf059e-ee3c-4f69-897b-452bf3d9d031 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:45.181054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8cf059e-ee3c-4f69-897b-452bf3d9d031 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:45.182774 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8cf059e-ee3c-4f69-897b-452bf3d9d031 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:45.201362 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 453/895] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:45.202630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cf059e-ee3c-4f69-897b-452bf3d9d031 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:45.234648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45858d80-778c-497a-b992-3927457a232f None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:45.275035 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42985778-6176-44a3-b808-5a76e1081f72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:45.275648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42985778-6176-44a3-b808-5a76e1081f72 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:45.275929 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42985778-6176-44a3-b808-5a76e1081f72 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:45.276375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42985778-6176-44a3-b808-5a76e1081f72 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:45.277022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42985778-6176-44a3-b808-5a76e1081f72 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:45.277429 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42985778-6176-44a3-b808-5a76e1081f72 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:45.278898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42985778-6176-44a3-b808-5a76e1081f72 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:45.289153 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 443/896] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:47:45 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:45.290084 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42985778-6176-44a3-b808-5a76e1081f72 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:45.312599 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cf059e-ee3c-4f69-897b-452bf3d9d031 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:45.349560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b170ace5-f3d9-48ce-a17f-d546f38f4184 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:45.350198 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b170ace5-f3d9-48ce-a17f-d546f38f4184 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:45.350480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b170ace5-f3d9-48ce-a17f-d546f38f4184 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:45.351386 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b170ace5-f3d9-48ce-a17f-d546f38f4184 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:45.357720 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b170ace5-f3d9-48ce-a17f-d546f38f4184 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:45.359166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b170ace5-f3d9-48ce-a17f-d546f38f4184 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:45.370783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-b170ace5-f3d9-48ce-a17f-d546f38f4184 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:45.370783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:45.370783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:45.370783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:45.370783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:45.370783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:45.370783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:45.370783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:45.370783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:45.370783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:45.370783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:45.370783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:45.370783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:45.370783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:45.370783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:45.371708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:45.371708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:45.371708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:45.371708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:45.371708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:45.371708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:45.371708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:45.371708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:45.371708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:45.371708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:45.371708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:45.371708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:45.371708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:45.371708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:45.371708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:45.371708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:45.371708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:45.372598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:45.372598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:45.372598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:45.372598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:45.372598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:45.372598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:45.372598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:45.372598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:45.372598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 454/897] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:45.374004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafac795-f742-47df-a224-a9877dcfac56 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:45.374004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafac795-f742-47df-a224-a9877dcfac56 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:45.374004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fafac795-f742-47df-a224-a9877dcfac56 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:45.377109 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 455/898] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:45.378427 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafac795-f742-47df-a224-a9877dcfac56 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:45.391924 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:45.403420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:45.407353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42985778-6176-44a3-b808-5a76e1081f72 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:45.446994 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca600b9-b68b-4cb6-b9c8-ccd4d0464aae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:45.447927 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca600b9-b68b-4cb6-b9c8-ccd4d0464aae None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:45.448544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca600b9-b68b-4cb6-b9c8-ccd4d0464aae None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:45.449016 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca600b9-b68b-4cb6-b9c8-ccd4d0464aae None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:45.449789 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca600b9-b68b-4cb6-b9c8-ccd4d0464aae None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:45.450092 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca600b9-b68b-4cb6-b9c8-ccd4d0464aae None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:45.451566 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca600b9-b68b-4cb6-b9c8-ccd4d0464aae None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:45.457725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 444/899] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:47:45 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:45.461152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a9bdc2-31b3-46ec-9c2c-8f335ca0bf5e None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:45.461152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a9bdc2-31b3-46ec-9c2c-8f335ca0bf5e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:45.461152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a9bdc2-31b3-46ec-9c2c-8f335ca0bf5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:45.479304 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-16a9bdc2-31b3-46ec-9c2c-8f335ca0bf5e None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:45.480500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-16a9bdc2-31b3-46ec-9c2c-8f335ca0bf5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:45.502347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fafac795-f742-47df-a224-a9877dcfac56 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:45.537790 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92feeac8-639d-4aa4-bfd8-7c80d53ac041 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:45.538393 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92feeac8-639d-4aa4-bfd8-7c80d53ac041 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:45.538668 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92feeac8-639d-4aa4-bfd8-7c80d53ac041 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:45.538939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92feeac8-639d-4aa4-bfd8-7c80d53ac041 None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:45.539624 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92feeac8-639d-4aa4-bfd8-7c80d53ac041 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:45.539928 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92feeac8-639d-4aa4-bfd8-7c80d53ac041 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:45.541368 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92feeac8-639d-4aa4-bfd8-7c80d53ac041 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:45.558537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 456/900] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:45.558537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92feeac8-639d-4aa4-bfd8-7c80d53ac041 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:45.561941 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 445/901] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:45 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:45.562900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a9bdc2-31b3-46ec-9c2c-8f335ca0bf5e None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:45.635205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:45.669333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92feeac8-639d-4aa4-bfd8-7c80d53ac041 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:45.690311 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:45.699056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a9bdc2-31b3-46ec-9c2c-8f335ca0bf5e None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:45.707679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd0fa999-310d-4646-aeb7-ef41f387adfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:45.708290 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0fa999-310d-4646-aeb7-ef41f387adfc None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:45.708569 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0fa999-310d-4646-aeb7-ef41f387adfc None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:45.708839 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd0fa999-310d-4646-aeb7-ef41f387adfc None admin] PATH_INFO: `/v3/endpoints` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:45.709574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0fa999-310d-4646-aeb7-ef41f387adfc None admin] RBAC: Adding query filter params (interface=public, service_id=003bd27ec8b44194a52d4e47c30801a2, region_id=RegionOne) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:45.709884 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0fa999-310d-4646-aeb7-ef41f387adfc None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:45.711246 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd0fa999-310d-4646-aeb7-ef41f387adfc None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:45.721278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 457/902] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Sep 4 00:47:45 2022] GET /identity/v3/endpoints?service_id=003bd27ec8b44194a52d4e47c30801a2&interface=public®ion_id=RegionOne => generated 194 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:45.722655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c88fce-9358-46ab-ad02-ea34253f67c0 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:45.723045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c88fce-9358-46ab-ad02-ea34253f67c0 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:45.723394 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c88fce-9358-46ab-ad02-ea34253f67c0 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:45.724693 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 458/903] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:45.726013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c88fce-9358-46ab-ad02-ea34253f67c0 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:45.740589 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a13f86-35a2-41cc-8725-2b2f446c0c4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:45.741238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a13f86-35a2-41cc-8725-2b2f446c0c4f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:45.741520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a13f86-35a2-41cc-8725-2b2f446c0c4f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:45.741795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a13f86-35a2-41cc-8725-2b2f446c0c4f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:45.742529 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a13f86-35a2-41cc-8725-2b2f446c0c4f None admin] RBAC: Adding query filter params (role.id=ace99e88214440f19edb734e93392300, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=70cef7c271e046a3b49230057cc2d28f) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:45.742842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a13f86-35a2-41cc-8725-2b2f446c0c4f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:45.744259 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a13f86-35a2-41cc-8725-2b2f446c0c4f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:45.751372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 446/904] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:45 2022] GET /identity/v3/role_assignments?user.id=70cef7c271e046a3b49230057cc2d28f&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=ace99e88214440f19edb734e93392300 => generated 259 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:45.753029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae64809-db6d-4bde-a4e4-9203229aefb6 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:45.753425 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae64809-db6d-4bde-a4e4-9203229aefb6 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:45.753809 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae64809-db6d-4bde-a4e4-9203229aefb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:45.781172 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-eae64809-db6d-4bde-a4e4-9203229aefb6 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:45.783827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-eae64809-db6d-4bde-a4e4-9203229aefb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:45.858610 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c88fce-9358-46ab-ad02-ea34253f67c0 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:45.869897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 447/905] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:45 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:45.870685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae64809-db6d-4bde-a4e4-9203229aefb6 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:45.908984 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb0476c-6c36-4993-95b1-fe0bc4ef01fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:45.909679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb0476c-6c36-4993-95b1-fe0bc4ef01fb None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:45.909957 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb0476c-6c36-4993-95b1-fe0bc4ef01fb None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:45.910222 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb0476c-6c36-4993-95b1-fe0bc4ef01fb None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:45.910825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb0476c-6c36-4993-95b1-fe0bc4ef01fb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:45.912182 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb0476c-6c36-4993-95b1-fe0bc4ef01fb None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:45.913446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:45.924070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 459/906] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 4 00:47:45 2022] GET /identity/v3/services => generated 1268 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 00:47:45.988781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:45.993475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae64809-db6d-4bde-a4e4-9203229aefb6 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:46.032355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d6b08c-406f-4ce2-84ee-71b4594be497 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:46.033573 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d6b08c-406f-4ce2-84ee-71b4594be497 None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:46.033939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d6b08c-406f-4ce2-84ee-71b4594be497 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:46.034188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d6b08c-406f-4ce2-84ee-71b4594be497 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:46.034848 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d6b08c-406f-4ce2-84ee-71b4594be497 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:46.036217 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d6b08c-406f-4ce2-84ee-71b4594be497 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:46.091254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 448/907] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 4 00:47:45 2022] POST /identity/v3/services => generated 241 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:47:46.152084 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb0476c-6c36-4993-95b1-fe0bc4ef01fb None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:46.268757 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb0476c-6c36-4993-95b1-fe0bc4ef01fb None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:46.312840 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d88280-dd0d-4d00-b9d7-08943560ae8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:46.313566 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d88280-dd0d-4d00-b9d7-08943560ae8c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:46.313847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d88280-dd0d-4d00-b9d7-08943560ae8c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:46.315150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d88280-dd0d-4d00-b9d7-08943560ae8c None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:46.315742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d88280-dd0d-4d00-b9d7-08943560ae8c None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:46.317131 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d88280-dd0d-4d00-b9d7-08943560ae8c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:46.326244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-62d88280-dd0d-4d00-b9d7-08943560ae8c None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 04 00:47:46.326244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:46.326244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:46.326244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:46.326244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:46.326244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:46.326244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:46.326244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:46.326244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 04 00:47:46.326244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:46.326244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:46.326244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:46.326244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:46.326244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:46.326244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:46.326244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:46.327154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:46.327154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:46.327154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:46.327154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:46.327154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:46.327154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:46.327154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:46.327154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:46.327154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:46.327154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:46.327154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:46.327154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:46.327154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:46.327154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:46.327154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:46.327154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:46.327154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:46.328022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:46.328022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:46.328022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:46.328022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:46.328022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:46.328022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:46.328022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:46.328022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:46.328022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:46.328022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:46.328022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:46.328022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:46.328022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 04 00:47:46.328022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:46.328022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 460/908] 10.222.0.42 () {56 vars in 1132 bytes} [Sun Sep 4 00:47:46 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:46.335986 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d6b08c-406f-4ce2-84ee-71b4594be497 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:46.444740 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d6b08c-406f-4ce2-84ee-71b4594be497 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:46.481436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea03b3b-0b7a-46e6-aefa-dc1e2eabd19f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:46.482052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea03b3b-0b7a-46e6-aefa-dc1e2eabd19f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:46.482336 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea03b3b-0b7a-46e6-aefa-dc1e2eabd19f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:46.482613 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea03b3b-0b7a-46e6-aefa-dc1e2eabd19f None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:46.483275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea03b3b-0b7a-46e6-aefa-dc1e2eabd19f None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:46.483583 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea03b3b-0b7a-46e6-aefa-dc1e2eabd19f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:46.485013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea03b3b-0b7a-46e6-aefa-dc1e2eabd19f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:46.491359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 449/909] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 4 00:47:46 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:46.498402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d88280-dd0d-4d00-b9d7-08943560ae8c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:46.558555 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:46.607392 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d88280-dd0d-4d00-b9d7-08943560ae8c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:46.644439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83b39bc-9ee6-41cb-86ab-8152965c6117 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:46.645039 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83b39bc-9ee6-41cb-86ab-8152965c6117 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:46.645343 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83b39bc-9ee6-41cb-86ab-8152965c6117 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:46.645615 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83b39bc-9ee6-41cb-86ab-8152965c6117 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:46.646249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83b39bc-9ee6-41cb-86ab-8152965c6117 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:46.646549 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83b39bc-9ee6-41cb-86ab-8152965c6117 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:46.647903 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83b39bc-9ee6-41cb-86ab-8152965c6117 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:46.653289 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 461/910] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 4 00:47:46 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:46.661901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea03b3b-0b7a-46e6-aefa-dc1e2eabd19f None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:46.679161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:46.805752 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea03b3b-0b7a-46e6-aefa-dc1e2eabd19f None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:46.846980 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410a78ec-6650-4cde-b46c-88b8fbbb9dc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:46.847607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410a78ec-6650-4cde-b46c-88b8fbbb9dc7 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:46.847891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410a78ec-6650-4cde-b46c-88b8fbbb9dc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:46.848167 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410a78ec-6650-4cde-b46c-88b8fbbb9dc7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:46.848886 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410a78ec-6650-4cde-b46c-88b8fbbb9dc7 None admin] RBAC: Adding query filter params (interface=public, service_id=8fb0fa24895c4570919449d02318e35d, region_id=RegionOne) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:46.849458 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410a78ec-6650-4cde-b46c-88b8fbbb9dc7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:46.850825 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410a78ec-6650-4cde-b46c-88b8fbbb9dc7 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:46.856426 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 450/911] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Sep 4 00:47:46 2022] GET /identity/v3/endpoints?service_id=8fb0fa24895c4570919449d02318e35d&interface=public®ion_id=RegionOne => generated 194 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:46.866750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83b39bc-9ee6-41cb-86ab-8152965c6117 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:47.023770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83b39bc-9ee6-41cb-86ab-8152965c6117 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:47.050159 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:47.069605 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254fff80-b090-41e7-beec-f3f824945b91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:47.070218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254fff80-b090-41e7-beec-f3f824945b91 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:47.070498 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254fff80-b090-41e7-beec-f3f824945b91 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:47.077261 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254fff80-b090-41e7-beec-f3f824945b91 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:47.077261 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254fff80-b090-41e7-beec-f3f824945b91 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:47.077261 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254fff80-b090-41e7-beec-f3f824945b91 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:47.092241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 462/912] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 4 00:47:46 2022] GET /identity/v3/services => generated 1497 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 00:47:47.916591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68bd4e4-63f1-4480-88b4-29afefa76f33 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:47.916591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68bd4e4-63f1-4480-88b4-29afefa76f33 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:47.916591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68bd4e4-63f1-4480-88b4-29afefa76f33 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:47.917777 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 451/913] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:47.924200 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12120b29-1a46-48e1-88c2-217cb0ce874c None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:47.925120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12120b29-1a46-48e1-88c2-217cb0ce874c None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:47.925654 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12120b29-1a46-48e1-88c2-217cb0ce874c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:47.945325 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-12120b29-1a46-48e1-88c2-217cb0ce874c None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:47.946823 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-12120b29-1a46-48e1-88c2-217cb0ce874c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:48.005924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07ff9be-30f0-41f3-ac6d-d0119ac5498d None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:48.006253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07ff9be-30f0-41f3-ac6d-d0119ac5498d None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:48.006463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07ff9be-30f0-41f3-ac6d-d0119ac5498d None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:48.008037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 452/914] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:47:48.013019 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46cf423-682e-4db8-8800-4e6a08a6bde0 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:48.013330 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46cf423-682e-4db8-8800-4e6a08a6bde0 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:48.013605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46cf423-682e-4db8-8800-4e6a08a6bde0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:48.025360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 463/915] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:47 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:48.033274 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-d46cf423-682e-4db8-8800-4e6a08a6bde0 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:48.034473 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d46cf423-682e-4db8-8800-4e6a08a6bde0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:48.103839 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 453/916] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:48 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:47:48.292693 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb739a14-1d43-4626-a5c9-d6fbe0881bd2 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:48.293001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb739a14-1d43-4626-a5c9-d6fbe0881bd2 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:48.293310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb739a14-1d43-4626-a5c9-d6fbe0881bd2 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:48.295024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 464/917] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:48.296214 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46cf423-682e-4db8-8800-4e6a08a6bde0 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:48.301685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd888acd-330d-49ca-a606-10cf850ed595 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:48.301852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:48.302890 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd888acd-330d-49ca-a606-10cf850ed595 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:48.303170 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd888acd-330d-49ca-a606-10cf850ed595 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:48.326358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-fd888acd-330d-49ca-a606-10cf850ed595 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:48.327739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-fd888acd-330d-49ca-a606-10cf850ed595 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:48.411702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46cf423-682e-4db8-8800-4e6a08a6bde0 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:48.418354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 465/918] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:48 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:47:48.419856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd888acd-330d-49ca-a606-10cf850ed595 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:48.457231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b814466b-df10-4b95-b704-dffbf9eec0d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:48.458778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b814466b-df10-4b95-b704-dffbf9eec0d5 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:48.459377 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b814466b-df10-4b95-b704-dffbf9eec0d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:48.459851 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b814466b-df10-4b95-b704-dffbf9eec0d5 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:48.475317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b814466b-df10-4b95-b704-dffbf9eec0d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:48.480319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-b814466b-df10-4b95-b704-dffbf9eec0d5 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 04 00:47:48.480319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:48.480319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:48.480319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:48.480319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:48.480319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:48.480319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:48.480319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:48.480319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:48.480319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:48.480319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:48.480319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:48.480319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:48.480319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:48.480319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:48.480319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:48.481355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:48.481355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:48.481355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:48.481355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:48.481355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:48.481355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:48.481355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:48.481355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:48.481355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:48.481355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:48.481355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:48.481355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:48.481355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:48.481355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:48.481355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:48.481355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:48.482263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:48.482263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:48.482263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:48.482263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:48.482263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:48.482263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:48.482263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:48.482263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:48.482263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:48.482263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:48.482263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 04 00:47:48.482263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:48.485222 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 454/919] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Sep 4 00:47:48 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:48.488451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b814466b-df10-4b95-b704-dffbf9eec0d5 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:48.498356 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:48.557362 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd888acd-330d-49ca-a606-10cf850ed595 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:48.570426 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:48.603749 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6031919f-6683-4aab-9dd8-ea72fbbfd78a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:48.604402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6031919f-6683-4aab-9dd8-ea72fbbfd78a None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:48.604745 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6031919f-6683-4aab-9dd8-ea72fbbfd78a None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:48.604950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6031919f-6683-4aab-9dd8-ea72fbbfd78a None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:48.605587 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6031919f-6683-4aab-9dd8-ea72fbbfd78a None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:48.606970 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6031919f-6683-4aab-9dd8-ea72fbbfd78a None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:48.613687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b814466b-df10-4b95-b704-dffbf9eec0d5 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:48.618117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-6031919f-6683-4aab-9dd8-ea72fbbfd78a None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 04 00:47:48.618117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:48.618117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:48.618117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:48.618117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:48.618117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:48.618117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:48.618117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:48.618117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 04 00:47:48.618117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:48.618117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:48.618117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:48.618117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:48.618117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:48.618117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:48.618117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:48.618117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:48.619122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:48.619122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:48.619122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:48.619122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:48.619122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:48.619122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:48.619122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:48.619122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:48.619122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:48.619122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:48.619122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:48.619122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:48.619122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:48.619122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:48.619122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:48.619122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:48.619956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:48.619956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:48.619956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:48.619956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:48.619956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:48.619956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:48.619956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:48.619956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:48.619956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:48.619956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:48.619956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:48.619956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:48.619956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 04 00:47:48.619956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:48.619956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 466/920] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:47:48 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:48.625476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6031919f-6683-4aab-9dd8-ea72fbbfd78a None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:48.655334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6dc240-ae22-4005-bdc7-c5b4b684cf1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:48.656310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6dc240-ae22-4005-bdc7-c5b4b684cf1d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:48.656803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6dc240-ae22-4005-bdc7-c5b4b684cf1d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:48.657411 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6dc240-ae22-4005-bdc7-c5b4b684cf1d None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:48.658413 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6dc240-ae22-4005-bdc7-c5b4b684cf1d None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:48.658940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6dc240-ae22-4005-bdc7-c5b4b684cf1d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:48.660586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6dc240-ae22-4005-bdc7-c5b4b684cf1d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:48.669877 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 455/921] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Sep 4 00:47:48 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:48.675726 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6dc240-ae22-4005-bdc7-c5b4b684cf1d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:48.758718 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6031919f-6683-4aab-9dd8-ea72fbbfd78a None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:48.773150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:48.800187 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6dc240-ae22-4005-bdc7-c5b4b684cf1d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:48.806457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ddecde-3244-47a8-8b07-0355b353833d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:48.807066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ddecde-3244-47a8-8b07-0355b353833d None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:48.807733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ddecde-3244-47a8-8b07-0355b353833d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:48.807733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ddecde-3244-47a8-8b07-0355b353833d None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:48.808237 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ddecde-3244-47a8-8b07-0355b353833d None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:48.808542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ddecde-3244-47a8-8b07-0355b353833d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:48.809956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ddecde-3244-47a8-8b07-0355b353833d None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:48.815417 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 467/922] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:47:48 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:48.816808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ddecde-3244-47a8-8b07-0355b353833d None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:48.827207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:48.837175 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c2ff22-1ef8-40bb-8c10-85bf32a38d35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:48.837819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c2ff22-1ef8-40bb-8c10-85bf32a38d35 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:48.843265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c2ff22-1ef8-40bb-8c10-85bf32a38d35 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:48.843265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c2ff22-1ef8-40bb-8c10-85bf32a38d35 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:48.849996 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c2ff22-1ef8-40bb-8c10-85bf32a38d35 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:48.851423 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c2ff22-1ef8-40bb-8c10-85bf32a38d35 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:48.866205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-00c2ff22-1ef8-40bb-8c10-85bf32a38d35 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:48.866205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:48.866205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:48.866205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:48.866205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:48.866205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:48.866205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:48.866205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:48.866205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:48.866205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:48.866205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:48.866205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:48.866205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:48.866205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:48.866205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:48.866205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:48.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:48.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:48.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:48.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:48.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:48.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:48.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:48.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:48.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:48.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:48.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:48.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:48.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:48.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:48.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:48.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:48.867284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:48.872813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:48.872813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:48.872813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:48.872813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 456/923] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:48.872813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c2ff22-1ef8-40bb-8c10-85bf32a38d35 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:48.931875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ddecde-3244-47a8-8b07-0355b353833d None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:48.968178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb13867-b932-4154-a0e6-7ae540a78387 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:48.968810 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb13867-b932-4154-a0e6-7ae540a78387 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:48.969098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb13867-b932-4154-a0e6-7ae540a78387 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:48.969390 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb13867-b932-4154-a0e6-7ae540a78387 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:48.969971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb13867-b932-4154-a0e6-7ae540a78387 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:48.972017 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb13867-b932-4154-a0e6-7ae540a78387 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:48.973969 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c2ff22-1ef8-40bb-8c10-85bf32a38d35 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:48.980711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-9eb13867-b932-4154-a0e6-7ae540a78387 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 04 00:47:48.980711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:48.980711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:48.980711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:48.980711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:48.980711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:48.980711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:48.980711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:48.980711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 04 00:47:48.980711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:48.980711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:48.980711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:48.980711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:48.980711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:48.980711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:48.980711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:48.980711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:48.981841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:48.981841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:48.981841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:48.981841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:48.981841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:48.981841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:48.981841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:48.981841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:48.981841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:48.981841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:48.981841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:48.981841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:48.981841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:48.981841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:48.981841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:48.981841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:48.982853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:48.982853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:48.982853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:48.982853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:48.982853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:48.982853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:48.982853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:48.982853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:48.982853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:48.982853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:48.982853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:48.982853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:48.982853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 04 00:47:48.982853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:48.986945 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 468/924] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 4 00:47:48 2022] GET /identity/v3/services/object-store => generated 93 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:48.987796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb13867-b932-4154-a0e6-7ae540a78387 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:49.014206 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23686709-48fb-410e-8bb6-78291a4c7ac2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:49.015112 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23686709-48fb-410e-8bb6-78291a4c7ac2 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:49.015550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23686709-48fb-410e-8bb6-78291a4c7ac2 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:49.016006 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23686709-48fb-410e-8bb6-78291a4c7ac2 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:49.016809 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23686709-48fb-410e-8bb6-78291a4c7ac2 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:49.017871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23686709-48fb-410e-8bb6-78291a4c7ac2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:49.019438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23686709-48fb-410e-8bb6-78291a4c7ac2 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:49.025380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 457/925] 10.222.0.42 () {56 vars in 1114 bytes} [Sun Sep 4 00:47:48 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:49.026346 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23686709-48fb-410e-8bb6-78291a4c7ac2 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:49.029626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:49.082123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:49.098509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb13867-b932-4154-a0e6-7ae540a78387 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:49.136361 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effb192d-1783-4831-8d7e-486445c634de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:49.136965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effb192d-1783-4831-8d7e-486445c634de None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:49.137268 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effb192d-1783-4831-8d7e-486445c634de None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:49.137539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effb192d-1783-4831-8d7e-486445c634de None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:49.138225 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effb192d-1783-4831-8d7e-486445c634de None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:49.138475 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effb192d-1783-4831-8d7e-486445c634de None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:49.139842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effb192d-1783-4831-8d7e-486445c634de None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:49.151714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 469/926] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:49.152218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23686709-48fb-410e-8bb6-78291a4c7ac2 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:49.152489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effb192d-1783-4831-8d7e-486445c634de None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:49.192711 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b53b490-f3c4-4f6f-b76a-255ccaab4121 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:49.193417 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b53b490-f3c4-4f6f-b76a-255ccaab4121 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:49.193705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b53b490-f3c4-4f6f-b76a-255ccaab4121 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:49.193978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b53b490-f3c4-4f6f-b76a-255ccaab4121 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:49.194612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b53b490-f3c4-4f6f-b76a-255ccaab4121 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:49.194921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b53b490-f3c4-4f6f-b76a-255ccaab4121 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:49.196389 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b53b490-f3c4-4f6f-b76a-255ccaab4121 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:49.205133 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 458/927] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 4 00:47:49 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:49.205133 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b53b490-f3c4-4f6f-b76a-255ccaab4121 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:49.260701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:49.264107 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effb192d-1783-4831-8d7e-486445c634de None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:49.297674 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bda305a-0f14-4a9e-8428-7db3b19244e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:49.298930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bda305a-0f14-4a9e-8428-7db3b19244e9 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:49.299377 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bda305a-0f14-4a9e-8428-7db3b19244e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:49.299658 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bda305a-0f14-4a9e-8428-7db3b19244e9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:49.300319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bda305a-0f14-4a9e-8428-7db3b19244e9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:49.301724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bda305a-0f14-4a9e-8428-7db3b19244e9 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:49.305489 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b53b490-f3c4-4f6f-b76a-255ccaab4121 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:49.312373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:49.337925 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facc90f1-2e46-40f9-b262-68c8d984fc2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:49.338484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facc90f1-2e46-40f9-b262-68c8d984fc2b None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:49.338755 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facc90f1-2e46-40f9-b262-68c8d984fc2b None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:49.339020 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facc90f1-2e46-40f9-b262-68c8d984fc2b None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:49.346237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facc90f1-2e46-40f9-b262-68c8d984fc2b None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:49.347639 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facc90f1-2e46-40f9-b262-68c8d984fc2b None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:49.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-facc90f1-2e46-40f9-b262-68c8d984fc2b None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 04 00:47:49.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:49.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:49.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:49.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:49.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:49.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:49.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:49.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:49.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:49.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:49.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:49.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:49.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:49.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:49.358646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:49.359695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:49.361380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:49.361380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:49.361380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:49.361380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:49.361380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:49.361380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:49.361380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:49.361380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:49.361380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 04 00:47:49.361380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:49.361380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 459/928] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 4 00:47:49 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:49.362121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facc90f1-2e46-40f9-b262-68c8d984fc2b None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:49.368842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 470/929] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 00:47:49 2022] POST /identity/v3/endpoints => generated 346 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:47:49.370151 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9082dbfd-2dc8-4e80-85bd-5a755cbb35b8 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:49.370441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9082dbfd-2dc8-4e80-85bd-5a755cbb35b8 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:49.370746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9082dbfd-2dc8-4e80-85bd-5a755cbb35b8 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:49.371951 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 471/930] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:49.372662 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9082dbfd-2dc8-4e80-85bd-5a755cbb35b8 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:49.498019 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facc90f1-2e46-40f9-b262-68c8d984fc2b None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:49.508745 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9082dbfd-2dc8-4e80-85bd-5a755cbb35b8 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:49.538043 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ce69fb-0163-4f58-9048-165c38c21317 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:49.538764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ce69fb-0163-4f58-9048-165c38c21317 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:49.539106 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ce69fb-0163-4f58-9048-165c38c21317 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:49.539384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ce69fb-0163-4f58-9048-165c38c21317 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:49.540024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ce69fb-0163-4f58-9048-165c38c21317 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:49.540334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ce69fb-0163-4f58-9048-165c38c21317 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:49.541733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ce69fb-0163-4f58-9048-165c38c21317 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:49.548653 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 460/931] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 4 00:47:49 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:49.550113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680b86c5-7ec0-437c-897c-0e31bf260086 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:49.550471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680b86c5-7ec0-437c-897c-0e31bf260086 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:49.550818 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680b86c5-7ec0-437c-897c-0e31bf260086 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:49.555180 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee571d28-dade-4506-820f-ec61a1989e51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:49.555894 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee571d28-dade-4506-820f-ec61a1989e51 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:49.556272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee571d28-dade-4506-820f-ec61a1989e51 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:49.556663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee571d28-dade-4506-820f-ec61a1989e51 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:49.557525 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee571d28-dade-4506-820f-ec61a1989e51 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:49.557927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee571d28-dade-4506-820f-ec61a1989e51 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:49.559445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee571d28-dade-4506-820f-ec61a1989e51 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:49.569106 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-680b86c5-7ec0-437c-897c-0e31bf260086 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:49.570322 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-680b86c5-7ec0-437c-897c-0e31bf260086 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:49.578304 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 472/932] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Sep 4 00:47:49 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:49.579140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee571d28-dade-4506-820f-ec61a1989e51 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:49.594088 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:49.601435 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:49.647835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 461/933] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:49 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:49.648688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680b86c5-7ec0-437c-897c-0e31bf260086 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:49.703767 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee571d28-dade-4506-820f-ec61a1989e51 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:49.742278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94237714-61c8-409f-bff9-551c814dce6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:49.743170 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94237714-61c8-409f-bff9-551c814dce6a None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:49.743596 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94237714-61c8-409f-bff9-551c814dce6a None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:49.744021 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94237714-61c8-409f-bff9-551c814dce6a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:49.744949 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94237714-61c8-409f-bff9-551c814dce6a None admin] RBAC: Adding query filter params (interface=public, service_id=128f48906f2b459f97dd52bd364518e5, region_id=RegionOne) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:49.745437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94237714-61c8-409f-bff9-551c814dce6a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:49.747061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94237714-61c8-409f-bff9-551c814dce6a None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:49.757451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 473/934] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Sep 4 00:47:49 2022] GET /identity/v3/endpoints?service_id=128f48906f2b459f97dd52bd364518e5&interface=public®ion_id=RegionOne => generated 194 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:49.759556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94237714-61c8-409f-bff9-551c814dce6a None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:49.774111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680b86c5-7ec0-437c-897c-0e31bf260086 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:49.819095 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a687e1-4daa-495c-b2c0-277f055c86d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:49.820026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a687e1-4daa-495c-b2c0-277f055c86d1 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:49.820313 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a687e1-4daa-495c-b2c0-277f055c86d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:49.820586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a687e1-4daa-495c-b2c0-277f055c86d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:49.835397 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a687e1-4daa-495c-b2c0-277f055c86d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:49.837432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a687e1-4daa-495c-b2c0-277f055c86d1 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:49.839031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:49.857000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-c6a687e1-4daa-495c-b2c0-277f055c86d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:49.857000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:49.857000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:49.857000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:49.857000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:49.857000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:49.857000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:49.857000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:49.857000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:49.857000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:49.857000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:49.857000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:49.857000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:49.857000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:49.857000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:49.857000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:49.858205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:49.858205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:49.858205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:49.858205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:49.858205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:49.858205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:49.858205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:49.858205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:49.858205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:49.858205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:49.858205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:49.858205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:49.858205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:49.858205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:49.858205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:49.858205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:49.858205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:49.859279 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:49.859279 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:49.859279 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:49.860543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 462/935] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:49.865891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a687e1-4daa-495c-b2c0-277f055c86d1 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:49.879865 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94237714-61c8-409f-bff9-551c814dce6a None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:49.887069 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:49.916935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7379882-3516-4f55-bb47-8e88c569c426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:49.917715 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7379882-3516-4f55-bb47-8e88c569c426 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:49.918086 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7379882-3516-4f55-bb47-8e88c569c426 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:49.918452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7379882-3516-4f55-bb47-8e88c569c426 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:49.919157 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7379882-3516-4f55-bb47-8e88c569c426 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:49.920605 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7379882-3516-4f55-bb47-8e88c569c426 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:49.929115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 474/936] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 4 00:47:49 2022] GET /identity/v3/services => generated 1497 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 00:47:49.960161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7379882-3516-4f55-bb47-8e88c569c426 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:50.003558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a687e1-4daa-495c-b2c0-277f055c86d1 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:50.042271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6e70a3-1f1e-4599-a3c8-f4859a6c7100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:50.042899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e70a3-1f1e-4599-a3c8-f4859a6c7100 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:50.043176 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e70a3-1f1e-4599-a3c8-f4859a6c7100 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:50.043449 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e70a3-1f1e-4599-a3c8-f4859a6c7100 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:50.044370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6e70a3-1f1e-4599-a3c8-f4859a6c7100 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:50.044497 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6e70a3-1f1e-4599-a3c8-f4859a6c7100 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:50.045835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6e70a3-1f1e-4599-a3c8-f4859a6c7100 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:50.061059 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 463/937] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:50.067192 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6e70a3-1f1e-4599-a3c8-f4859a6c7100 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:50.108595 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7379882-3516-4f55-bb47-8e88c569c426 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:50.145297 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21c6710-3480-4dd6-a815-c6f974296f7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:50.145932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21c6710-3480-4dd6-a815-c6f974296f7d None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:50.146207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21c6710-3480-4dd6-a815-c6f974296f7d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:50.146476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21c6710-3480-4dd6-a815-c6f974296f7d None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:50.147052 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21c6710-3480-4dd6-a815-c6f974296f7d None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:50.148466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21c6710-3480-4dd6-a815-c6f974296f7d None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:50.152180 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:50.158791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:50.159622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-a21c6710-3480-4dd6-a815-c6f974296f7d None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 04 00:47:50.159622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:50.159622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:50.159622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:50.159622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:50.159622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:50.159622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:50.159622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:50.159622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 04 00:47:50.159622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:50.159622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:50.159622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:50.159622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:50.159622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:50.159622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:50.159622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:50.160517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:50.160517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:50.160517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:50.160517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:50.160517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:50.160517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:50.160517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:50.160517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:50.160517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:50.160517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:50.160517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:50.160517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:50.160517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:50.160517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:50.160517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:50.160517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:50.160517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:50.161438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:50.161438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:50.161438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:50.161438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:50.161438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:50.161438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:50.161438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:50.161438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:50.161438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:50.161438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:50.161438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:50.161438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:50.161438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 04 00:47:50.161438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:50.165160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 475/938] 10.222.0.42 () {56 vars in 1132 bytes} [Sun Sep 4 00:47:49 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:50.171087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21c6710-3480-4dd6-a815-c6f974296f7d None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:50.194581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6e70a3-1f1e-4599-a3c8-f4859a6c7100 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:50.242075 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52877923-5ace-4dfd-8167-9ab2b187201b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:50.242705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52877923-5ace-4dfd-8167-9ab2b187201b None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:50.242983 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52877923-5ace-4dfd-8167-9ab2b187201b None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:50.243253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52877923-5ace-4dfd-8167-9ab2b187201b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:50.250182 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52877923-5ace-4dfd-8167-9ab2b187201b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:50.252209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52877923-5ace-4dfd-8167-9ab2b187201b None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:50.264586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-52877923-5ace-4dfd-8167-9ab2b187201b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:50.264586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:50.264586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:50.264586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:50.264586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:50.264586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:50.264586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:50.264586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:50.264586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:50.264586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:50.264586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:50.264586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:50.264586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:50.264586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:50.264586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:50.265571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:50.265571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:50.265571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:50.265571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:50.265571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:50.265571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:50.265571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:50.265571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:50.265571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:50.265571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:50.265571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:50.265571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:50.265571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:50.265571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:50.265571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:50.265571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:50.265571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:50.266462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:50.266462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:50.266462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:50.266462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:50.266462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:50.266462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:50.266462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:50.266462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:50.266462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 464/939] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:50.271767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52877923-5ace-4dfd-8167-9ab2b187201b None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:50.303763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21c6710-3480-4dd6-a815-c6f974296f7d None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:50.346838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f26de07-55dc-4df8-9699-1e0090e96db7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:50.347869 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f26de07-55dc-4df8-9699-1e0090e96db7 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:50.349046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f26de07-55dc-4df8-9699-1e0090e96db7 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:50.349426 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f26de07-55dc-4df8-9699-1e0090e96db7 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:50.350136 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f26de07-55dc-4df8-9699-1e0090e96db7 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:50.350511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f26de07-55dc-4df8-9699-1e0090e96db7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:50.351945 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f26de07-55dc-4df8-9699-1e0090e96db7 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:50.358271 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 476/940] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 4 00:47:50 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:50.366132 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f26de07-55dc-4df8-9699-1e0090e96db7 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:50.394967 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52877923-5ace-4dfd-8167-9ab2b187201b None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:50.444527 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:50.456133 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c0a5ee-c101-4b33-9ba6-e453cd608f91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:50.456750 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c0a5ee-c101-4b33-9ba6-e453cd608f91 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:50.457040 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c0a5ee-c101-4b33-9ba6-e453cd608f91 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:50.457362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c0a5ee-c101-4b33-9ba6-e453cd608f91 None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:50.460648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c0a5ee-c101-4b33-9ba6-e453cd608f91 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:50.460648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c0a5ee-c101-4b33-9ba6-e453cd608f91 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:50.460648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c0a5ee-c101-4b33-9ba6-e453cd608f91 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:50.474765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:50.481593 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 465/941] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:50.482870 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c0a5ee-c101-4b33-9ba6-e453cd608f91 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:50.531022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f26de07-55dc-4df8-9699-1e0090e96db7 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:50.581363 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0fdf9d-43c0-46e7-8df0-00261bd279c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:50.582488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0fdf9d-43c0-46e7-8df0-00261bd279c1 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:50.582921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0fdf9d-43c0-46e7-8df0-00261bd279c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:50.583626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0fdf9d-43c0-46e7-8df0-00261bd279c1 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:50.584858 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0fdf9d-43c0-46e7-8df0-00261bd279c1 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:50.585626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0fdf9d-43c0-46e7-8df0-00261bd279c1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:50.587801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0fdf9d-43c0-46e7-8df0-00261bd279c1 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:50.593776 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c0a5ee-c101-4b33-9ba6-e453cd608f91 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:50.594733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 477/942] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 4 00:47:50 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:50.600975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0fdf9d-43c0-46e7-8df0-00261bd279c1 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:50.643475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8d685a-eb9d-4f8c-bacd-7fe4ff988442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:50.644378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8d685a-eb9d-4f8c-bacd-7fe4ff988442 None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:50.644816 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8d685a-eb9d-4f8c-bacd-7fe4ff988442 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:50.645296 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8d685a-eb9d-4f8c-bacd-7fe4ff988442 None admin] PATH_INFO: `/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/70cef7c271e046a3b49230057cc2d28f/roles/ace99e88214440f19edb734e93392300` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:50.648201 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8d685a-eb9d-4f8c-bacd-7fe4ff988442 None admin] RBAC: Authorizing `identity:create_grant(project_id=b1f56314d44d4faaa88e735ea196783f, user_id=70cef7c271e046a3b49230057cc2d28f, role_id=ace99e88214440f19edb734e93392300)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:50.649843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8d685a-eb9d-4f8c-bacd-7fe4ff988442 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:50.715976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0fdf9d-43c0-46e7-8df0-00261bd279c1 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:50.749051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 466/943] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:47:50 2022] PUT /identity/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/70cef7c271e046a3b49230057cc2d28f/roles/ace99e88214440f19edb734e93392300 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:47:50.751914 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:50.760254 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413968f5-8e2f-4f53-b0c2-c1f90444ac86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:50.760883 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413968f5-8e2f-4f53-b0c2-c1f90444ac86 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:50.761326 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413968f5-8e2f-4f53-b0c2-c1f90444ac86 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:50.761607 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413968f5-8e2f-4f53-b0c2-c1f90444ac86 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:50.762405 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413968f5-8e2f-4f53-b0c2-c1f90444ac86 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:50.763953 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413968f5-8e2f-4f53-b0c2-c1f90444ac86 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:50.873630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 478/944] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 00:47:50 2022] POST /identity/v3/endpoints => generated 347 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 00:47:51.432020 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6e80df-9340-4cc5-b1d4-2bca19c1c656 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:51.432533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6e80df-9340-4cc5-b1d4-2bca19c1c656 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:51.432612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6e80df-9340-4cc5-b1d4-2bca19c1c656 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:51.434306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 467/945] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:51.439750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d6fa82-932f-4a1c-93d7-6fb68e33e61a None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:51.440155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d6fa82-932f-4a1c-93d7-6fb68e33e61a None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:51.440334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d6fa82-932f-4a1c-93d7-6fb68e33e61a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:51.461430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-72d6fa82-932f-4a1c-93d7-6fb68e33e61a None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:51.462722 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-72d6fa82-932f-4a1c-93d7-6fb68e33e61a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:51.543618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 479/946] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:51 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:51.827676 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6e80df-9340-4cc5-b1d4-2bca19c1c656 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:51.887438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:51.937713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6e80df-9340-4cc5-b1d4-2bca19c1c656 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:51.943345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de57a771-1743-4cac-88d6-a84a152db9f7 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:51.943628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de57a771-1743-4cac-88d6-a84a152db9f7 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:51.943906 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de57a771-1743-4cac-88d6-a84a152db9f7 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:51.945669 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 480/947] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:51.950653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73ae8a6-4ce0-49ab-88b4-909f51207ddc None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:51.950936 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73ae8a6-4ce0-49ab-88b4-909f51207ddc None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:51.951200 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73ae8a6-4ce0-49ab-88b4-909f51207ddc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:51.974124 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-b73ae8a6-4ce0-49ab-88b4-909f51207ddc None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:51.975313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b73ae8a6-4ce0-49ab-88b4-909f51207ddc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:51.977776 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5950e0c6-25ac-4691-b13f-3a5d0c14981d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:51.978692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5950e0c6-25ac-4691-b13f-3a5d0c14981d None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:51.979200 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5950e0c6-25ac-4691-b13f-3a5d0c14981d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:51.979682 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5950e0c6-25ac-4691-b13f-3a5d0c14981d None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:51.980553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5950e0c6-25ac-4691-b13f-3a5d0c14981d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:51.982249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5950e0c6-25ac-4691-b13f-3a5d0c14981d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:51.985370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: WARNING keystone.server.flask.common [None req-5950e0c6-25ac-4691-b13f-3a5d0c14981d None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 04 00:47:52.020749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.sql.core [None req-5950e0c6-25ac-4691-b13f-3a5d0c14981d None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 04 00:47:52.020749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 04 00:47:52.020749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [parameters: {'id': 'f1da5a392c804a369741f1ec325c335d', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 04 00:47:52.020749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81559) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 04 00:47:52.034852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-5950e0c6-25ac-4691-b13f-3a5d0c14981d None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id f1da5a392c804a369741f1ec325c335d.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id f1da5a392c804a369741f1ec325c335d. Sep 04 00:47:52.034852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:52.034852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 04 00:47:52.034852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 04 00:47:52.034852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 04 00:47:52.034852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 04 00:47:52.034852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 04 00:47:52.034852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application result = self._query(query) Sep 04 00:47:52.034852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 04 00:47:52.034852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application conn.query(q) Sep 04 00:47:52.034852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 04 00:47:52.034852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 04 00:47:52.034852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application result.read() Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 04 00:47:52.035790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 04 00:47:52.036725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 04 00:47:52.036725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application next(self.gen) Sep 04 00:47:52.036725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 04 00:47:52.036725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application yield resource Sep 04 00:47:52.036725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 04 00:47:52.036725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application next(self.gen) Sep 04 00:47:52.036725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 04 00:47:52.036725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self.session.rollback() Sep 04 00:47:52.036725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 04 00:47:52.036725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self.force_reraise() Sep 04 00:47:52.036725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 04 00:47:52.036725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise self.value Sep 04 00:47:52.036725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 04 00:47:52.036725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 04 00:47:52.036725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 04 00:47:52.036725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application session.commit() Sep 04 00:47:52.036725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 04 00:47:52.037631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 04 00:47:52.037631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 04 00:47:52.037631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._prepare_impl() Sep 04 00:47:52.037631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 04 00:47:52.037631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self.session.flush() Sep 04 00:47:52.037631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 04 00:47:52.037631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._flush(objects) Sep 04 00:47:52.037631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 04 00:47:52.037631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 04 00:47:52.037631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 04 00:47:52.037631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application compat.raise_( Sep 04 00:47:52.037631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 04 00:47:52.037631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception Sep 04 00:47:52.037631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 04 00:47:52.037631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application flush_context.execute() Sep 04 00:47:52.037631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 04 00:47:52.037631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rec.execute(self) Sep 04 00:47:52.038541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 04 00:47:52.038541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 04 00:47:52.038541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 04 00:47:52.038541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 04 00:47:52.038541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 04 00:47:52.038541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 04 00:47:52.038541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 04 00:47:52.038541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 04 00:47:52.038541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 04 00:47:52.038541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 04 00:47:52.038541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 04 00:47:52.038541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 04 00:47:52.038541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 04 00:47:52.038541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 04 00:47:52.038541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 04 00:47:52.039338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 04 00:47:52.039338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 04 00:47:52.039338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception Sep 04 00:47:52.039338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 04 00:47:52.039338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 04 00:47:52.039338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 04 00:47:52.039338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 04 00:47:52.039338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 04 00:47:52.039338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application result = self._query(query) Sep 04 00:47:52.039338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 04 00:47:52.039338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application conn.query(q) Sep 04 00:47:52.039338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 04 00:47:52.039338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 04 00:47:52.039338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 04 00:47:52.039338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application result.read() Sep 04 00:47:52.039338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 04 00:47:52.039338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 04 00:47:52.040203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 04 00:47:52.040203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 04 00:47:52.040203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 04 00:47:52.040203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 04 00:47:52.040203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 04 00:47:52.040203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 04 00:47:52.040203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 04 00:47:52.040203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 04 00:47:52.040203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [parameters: {'id': 'f1da5a392c804a369741f1ec325c335d', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 04 00:47:52.040203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 04 00:47:52.040203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:52.040203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:52.040203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:52.040203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:52.040203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 04 00:47:52.040998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 04 00:47:52.040998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 04 00:47:52.040998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 04 00:47:52.040998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 04 00:47:52.040998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:52.040998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:52.040998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:52.040998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:52.040998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:52.040998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:52.040998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:52.040998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:52.040998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:52.040998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:52.040998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:52.040998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:52.040998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:52.043940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:52.043940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 04 00:47:52.043940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 04 00:47:52.043940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:52.043940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:52.043940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 04 00:47:52.043940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 04 00:47:52.043940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id f1da5a392c804a369741f1ec325c335d. Sep 04 00:47:52.043940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:52.043940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 468/948] 10.222.0.42 () {60 vars in 1140 bytes} [Sun Sep 4 00:47:51 2022] POST /identity/v3/projects => generated 262 bytes in 212 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:52.051002 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5950e0c6-25ac-4691-b13f-3a5d0c14981d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:52.073108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 481/949] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:51 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:47:52.162535 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5950e0c6-25ac-4691-b13f-3a5d0c14981d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:52.172219 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:52.198160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cccbe7-9c01-473f-b561-5d377defbe56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:52.198847 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cccbe7-9c01-473f-b561-5d377defbe56 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:52.199127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cccbe7-9c01-473f-b561-5d377defbe56 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:52.199399 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cccbe7-9c01-473f-b561-5d377defbe56 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:52.207157 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cccbe7-9c01-473f-b561-5d377defbe56 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:52.208885 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cccbe7-9c01-473f-b561-5d377defbe56 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:52.221015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-47cccbe7-9c01-473f-b561-5d377defbe56 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:52.221015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:52.221015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:52.221015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:52.221015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:52.221015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:52.221015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:52.221015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:52.221015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:52.221015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:52.221015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:52.221015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:52.221015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:52.221015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:52.221015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:52.221939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:52.221939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:52.221939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:52.221939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:52.221939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:52.221939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:52.221939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:52.221939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:52.221939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:52.221939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:52.221939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:52.221939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:52.221939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:52.221939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:52.221939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:52.221939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:52.221939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:52.222835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:52.222835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:52.222835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:52.222835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:52.222835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:52.222835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:52.222835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:52.222835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:52.222835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 469/950] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:52.234162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73ae8a6-4ce0-49ab-88b4-909f51207ddc None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:52.280874 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:52.344285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73ae8a6-4ce0-49ab-88b4-909f51207ddc None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:52.369892 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cccbe7-9c01-473f-b561-5d377defbe56 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:52.387085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5abaf8-0275-4783-aafd-11ad1585fb33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:52.387701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5abaf8-0275-4783-aafd-11ad1585fb33 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:52.387980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5abaf8-0275-4783-aafd-11ad1585fb33 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:52.388253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5abaf8-0275-4783-aafd-11ad1585fb33 None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:52.389394 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5abaf8-0275-4783-aafd-11ad1585fb33 None admin] RBAC: Adding query filter params (name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:52.389699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5abaf8-0275-4783-aafd-11ad1585fb33 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:52.391526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5abaf8-0275-4783-aafd-11ad1585fb33 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:52.422201 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 482/951] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 4 00:47:52 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:52.512747 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cccbe7-9c01-473f-b561-5d377defbe56 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:52.552625 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbaf11a-3ede-48e2-859e-953873c87d18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:52.553312 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbaf11a-3ede-48e2-859e-953873c87d18 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:52.553597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbaf11a-3ede-48e2-859e-953873c87d18 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:52.553870 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbaf11a-3ede-48e2-859e-953873c87d18 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:52.554538 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbaf11a-3ede-48e2-859e-953873c87d18 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:52.555921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbaf11a-3ede-48e2-859e-953873c87d18 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:52.566681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-5cbaf11a-3ede-48e2-859e-953873c87d18 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 04 00:47:52.566681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:52.566681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:52.566681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:52.566681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:52.566681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:52.566681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:52.566681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:52.566681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 04 00:47:52.566681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:52.566681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:52.566681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:52.566681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:52.566681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:52.566681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:52.566681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:52.566681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:52.567852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:52.567852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:52.567852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:52.567852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:52.567852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:52.567852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:52.567852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:52.567852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:52.567852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:52.567852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:52.567852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:52.567852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:52.567852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:52.567852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:52.567852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:52.567852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:52.569576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:52.569576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:52.569576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:52.569576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:52.569576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:52.569576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:52.569576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:52.569576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:52.569576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:52.569576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:52.569576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:52.569576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:52.569576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 04 00:47:52.569576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:52.569576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 470/952] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 4 00:47:52 2022] GET /identity/v3/services/object-store => generated 93 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:52.574798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5abaf8-0275-4783-aafd-11ad1585fb33 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:52.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:52.683197 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5abaf8-0275-4783-aafd-11ad1585fb33 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:52.719575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6583ad-87e3-4c9c-9da1-a9c3e7cdf3e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:52.720849 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6583ad-87e3-4c9c-9da1-a9c3e7cdf3e9 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:52.721156 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6583ad-87e3-4c9c-9da1-a9c3e7cdf3e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:52.721524 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6583ad-87e3-4c9c-9da1-a9c3e7cdf3e9 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:52.722161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6583ad-87e3-4c9c-9da1-a9c3e7cdf3e9 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:52.722466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6583ad-87e3-4c9c-9da1-a9c3e7cdf3e9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:52.723828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6583ad-87e3-4c9c-9da1-a9c3e7cdf3e9 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:52.729191 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 483/953] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 4 00:47:52 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:52.738540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbaf11a-3ede-48e2-859e-953873c87d18 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:52.750505 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160faec8-6f0e-4d2f-8958-4336d37f74f0 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:52.750790 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160faec8-6f0e-4d2f-8958-4336d37f74f0 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:52.751057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160faec8-6f0e-4d2f-8958-4336d37f74f0 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:52.752687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 484/954] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:52.757578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6aea1fc-0c5a-44ec-adb9-8979e7e25e44 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:52.757859 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6aea1fc-0c5a-44ec-adb9-8979e7e25e44 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:52.758134 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6aea1fc-0c5a-44ec-adb9-8979e7e25e44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:52.774223 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:52.776820 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-e6aea1fc-0c5a-44ec-adb9-8979e7e25e44 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:52.778037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e6aea1fc-0c5a-44ec-adb9-8979e7e25e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:52.851240 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbaf11a-3ede-48e2-859e-953873c87d18 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:52.855345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 485/955] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:52 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:47:52.888182 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154af854-08cd-44a3-bcf5-c1334cdd2e29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:52.888665 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f0f1e7-8e00-47dd-8561-0c2318574e2e None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:52.888799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154af854-08cd-44a3-bcf5-c1334cdd2e29 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:52.889102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154af854-08cd-44a3-bcf5-c1334cdd2e29 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:52.889383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154af854-08cd-44a3-bcf5-c1334cdd2e29 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:52.889726 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f0f1e7-8e00-47dd-8561-0c2318574e2e None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:52.890026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154af854-08cd-44a3-bcf5-c1334cdd2e29 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:52.890327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154af854-08cd-44a3-bcf5-c1334cdd2e29 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:52.890634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f0f1e7-8e00-47dd-8561-0c2318574e2e None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:52.891689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154af854-08cd-44a3-bcf5-c1334cdd2e29 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:52.893462 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 486/956] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:52 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:47:52.899064 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afda148-d05e-4085-8644-dd885d1c6c86 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:52.899926 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afda148-d05e-4085-8644-dd885d1c6c86 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:52.900423 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afda148-d05e-4085-8644-dd885d1c6c86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:52.904438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 471/957] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 4 00:47:52 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:52.913158 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154af854-08cd-44a3-bcf5-c1334cdd2e29 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:52.923918 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-1afda148-d05e-4085-8644-dd885d1c6c86 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:52.925276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-1afda148-d05e-4085-8644-dd885d1c6c86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:52.999639 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 487/958] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:52 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:53.025939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:53.032176 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154af854-08cd-44a3-bcf5-c1334cdd2e29 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:53.070026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd75382-9ab2-439a-93ee-0f02c7d4b39a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:53.070693 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd75382-9ab2-439a-93ee-0f02c7d4b39a None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:53.070982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd75382-9ab2-439a-93ee-0f02c7d4b39a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:53.071261 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd75382-9ab2-439a-93ee-0f02c7d4b39a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:53.071929 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd75382-9ab2-439a-93ee-0f02c7d4b39a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:53.073328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd75382-9ab2-439a-93ee-0f02c7d4b39a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:53.132681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afda148-d05e-4085-8644-dd885d1c6c86 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:53.159141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:53.163452 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 472/959] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 00:47:52 2022] POST /identity/v3/endpoints => generated 349 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:47:53.252734 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afda148-d05e-4085-8644-dd885d1c6c86 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:53.269771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd75382-9ab2-439a-93ee-0f02c7d4b39a None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:53.303896 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2770bc9-a792-4b85-8c02-74d505cde856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:53.304935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2770bc9-a792-4b85-8c02-74d505cde856 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:53.305474 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2770bc9-a792-4b85-8c02-74d505cde856 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:53.305940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2770bc9-a792-4b85-8c02-74d505cde856 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:53.319204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2770bc9-a792-4b85-8c02-74d505cde856 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:47:53.326204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-e2770bc9-a792-4b85-8c02-74d505cde856 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 04 00:47:53.326204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:53.326204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:53.326204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:53.326204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:53.326204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:53.326204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:53.326204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:53.326204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:53.326204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:53.326204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:53.326204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:53.326204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:47:53.326204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:47:53.326204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:47:53.326204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:47:53.327221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:47:53.327221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:47:53.327221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:47:53.327221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:47:53.327221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:53.327221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:53.327221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:53.327221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:53.327221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:53.327221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:53.327221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:53.327221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:53.327221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:53.327221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:53.327221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:53.327221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:53.328030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:53.328030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:53.328030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:53.328030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:53.328030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:47:53.328030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:47:53.328030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:47:53.328030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:47:53.328030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:47:53.328030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:47:53.328030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 04 00:47:53.328030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:53.331004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 488/960] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Sep 4 00:47:53 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:53.337166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2770bc9-a792-4b85-8c02-74d505cde856 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:53.410309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd75382-9ab2-439a-93ee-0f02c7d4b39a None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:53.436490 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:53.450424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52fc8e5-36ef-4467-bf1e-6b797a9bcfe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:53.451409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52fc8e5-36ef-4467-bf1e-6b797a9bcfe6 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:53.451959 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52fc8e5-36ef-4467-bf1e-6b797a9bcfe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:53.452444 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52fc8e5-36ef-4467-bf1e-6b797a9bcfe6 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:53.453244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52fc8e5-36ef-4467-bf1e-6b797a9bcfe6 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:53.454943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52fc8e5-36ef-4467-bf1e-6b797a9bcfe6 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:53.462388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2770bc9-a792-4b85-8c02-74d505cde856 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:53.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-e52fc8e5-36ef-4467-bf1e-6b797a9bcfe6 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 04 00:47:53.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:53.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:53.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:53.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:53.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:53.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:53.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:53.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 04 00:47:53.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:53.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:53.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:53.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:53.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:53.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:53.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:53.466225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:53.467209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:53.467209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:53.467209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:53.467209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:53.467209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:53.467209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:53.467209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:53.467209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:53.467209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:53.467209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:53.467209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:53.467209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:53.467209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:53.467209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:53.467209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:53.467209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:53.468065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:53.468065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:53.468065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:53.468065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:53.468065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:53.468065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:53.468065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:53.468065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:53.468065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:53.468065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:53.468065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:53.468065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:53.468065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 04 00:47:53.468065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:53.471322 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 473/961] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:53 2022] GET /identity/v3/services/compute => generated 88 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:53.481197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52fc8e5-36ef-4467-bf1e-6b797a9bcfe6 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:53.498663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726d6977-c1e8-4383-8438-aab800fcf461 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:53.499269 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726d6977-c1e8-4383-8438-aab800fcf461 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:53.499549 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726d6977-c1e8-4383-8438-aab800fcf461 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:53.499876 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726d6977-c1e8-4383-8438-aab800fcf461 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:53.500487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726d6977-c1e8-4383-8438-aab800fcf461 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:53.500995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726d6977-c1e8-4383-8438-aab800fcf461 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:53.504098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726d6977-c1e8-4383-8438-aab800fcf461 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:53.506508 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:53.512807 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 489/962] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Sep 4 00:47:53 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:53.522653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726d6977-c1e8-4383-8438-aab800fcf461 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:53.606983 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52fc8e5-36ef-4467-bf1e-6b797a9bcfe6 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:53.650374 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726d6977-c1e8-4383-8438-aab800fcf461 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:53.651872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da1f749-3fe9-4b5c-893e-9bc73cc4589d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:53.652504 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da1f749-3fe9-4b5c-893e-9bc73cc4589d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:53.652782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da1f749-3fe9-4b5c-893e-9bc73cc4589d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:53.653050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da1f749-3fe9-4b5c-893e-9bc73cc4589d None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:53.654133 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da1f749-3fe9-4b5c-893e-9bc73cc4589d None admin] RBAC: Adding query filter params (name=compute) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:53.654437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da1f749-3fe9-4b5c-893e-9bc73cc4589d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:53.657046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da1f749-3fe9-4b5c-893e-9bc73cc4589d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:53.665125 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 474/963] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 4 00:47:53 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:53.671663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da1f749-3fe9-4b5c-893e-9bc73cc4589d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:53.696799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53c3f6e-f217-493f-b03b-e95531559e19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:53.697569 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53c3f6e-f217-493f-b03b-e95531559e19 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:53.697914 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53c3f6e-f217-493f-b03b-e95531559e19 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:53.698491 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53c3f6e-f217-493f-b03b-e95531559e19 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:53.710440 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53c3f6e-f217-493f-b03b-e95531559e19 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:53.711812 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53c3f6e-f217-493f-b03b-e95531559e19 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:53.713965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:53.727759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-c53c3f6e-f217-493f-b03b-e95531559e19 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:53.727759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:53.727759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:53.727759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:53.727759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:53.727759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:53.727759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:53.727759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:53.727759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:53.727759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:53.727759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:53.727759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:53.727759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:53.727759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:53.727759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:53.727759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:53.728742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:53.728742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:53.728742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:53.728742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:53.728742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:53.728742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:53.728742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:53.728742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:53.728742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:53.728742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:53.728742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:53.728742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:53.728742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:53.728742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:53.728742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:53.728742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:53.728742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:53.729678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:53.729678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:53.729678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:53.730453 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 490/964] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:53.736219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53c3f6e-f217-493f-b03b-e95531559e19 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:53.782210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:53.809064 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da1f749-3fe9-4b5c-893e-9bc73cc4589d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:53.845535 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53c3f6e-f217-493f-b03b-e95531559e19 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:53.857641 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c7f81d-2791-4a66-8716-3b0d9bd09e22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:53.858247 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c7f81d-2791-4a66-8716-3b0d9bd09e22 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:53.858525 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c7f81d-2791-4a66-8716-3b0d9bd09e22 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:53.858796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c7f81d-2791-4a66-8716-3b0d9bd09e22 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:53.859430 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c7f81d-2791-4a66-8716-3b0d9bd09e22 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:53.860529 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c7f81d-2791-4a66-8716-3b0d9bd09e22 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:53.862596 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c7f81d-2791-4a66-8716-3b0d9bd09e22 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:53.877612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 475/965] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 4 00:47:53 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:53.892743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52345ffe-eff8-49e6-8bb5-19cbceb0e192 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:53.893852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52345ffe-eff8-49e6-8bb5-19cbceb0e192 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:53.894335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52345ffe-eff8-49e6-8bb5-19cbceb0e192 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:53.894804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52345ffe-eff8-49e6-8bb5-19cbceb0e192 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:53.895742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52345ffe-eff8-49e6-8bb5-19cbceb0e192 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:53.896277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52345ffe-eff8-49e6-8bb5-19cbceb0e192 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:53.897978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52345ffe-eff8-49e6-8bb5-19cbceb0e192 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:53.914047 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 491/966] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:53.922627 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c7f81d-2791-4a66-8716-3b0d9bd09e22 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:54.037335 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c7f81d-2791-4a66-8716-3b0d9bd09e22 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:54.072664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50581ad-fafb-403c-b0c9-83bdd6272082 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:54.073764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50581ad-fafb-403c-b0c9-83bdd6272082 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:54.074341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50581ad-fafb-403c-b0c9-83bdd6272082 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:54.074824 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50581ad-fafb-403c-b0c9-83bdd6272082 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:54.083458 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50581ad-fafb-403c-b0c9-83bdd6272082 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:54.085179 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50581ad-fafb-403c-b0c9-83bdd6272082 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:54.097421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-d50581ad-fafb-403c-b0c9-83bdd6272082 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 04 00:47:54.097421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:54.097421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:54.097421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:54.097421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:54.097421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:54.097421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:54.097421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:54.097421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:54.097421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:54.097421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:54.097421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:54.097421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:54.097421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:54.097421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:54.097421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:54.098402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:54.099323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:54.099323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:54.099323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:54.099323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:54.099323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:54.099323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:54.099323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:54.099323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:54.099323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 04 00:47:54.099323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:54.101838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 476/967] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 4 00:47:53 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:54.107419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52345ffe-eff8-49e6-8bb5-19cbceb0e192 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:54.167151 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:54.215002 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52345ffe-eff8-49e6-8bb5-19cbceb0e192 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:54.252264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b13d68-5793-4388-bb84-0d61f0c37fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:54.253284 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b13d68-5793-4388-bb84-0d61f0c37fb9 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:54.253820 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b13d68-5793-4388-bb84-0d61f0c37fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:54.254303 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b13d68-5793-4388-bb84-0d61f0c37fb9 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:54.255265 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b13d68-5793-4388-bb84-0d61f0c37fb9 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:54.255785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b13d68-5793-4388-bb84-0d61f0c37fb9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:54.257440 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b13d68-5793-4388-bb84-0d61f0c37fb9 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:54.276586 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 492/968] 10.222.0.42 () {56 vars in 1149 bytes} [Sun Sep 4 00:47:54 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:54.282215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50581ad-fafb-403c-b0c9-83bdd6272082 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:54.287664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:54.402837 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50581ad-fafb-403c-b0c9-83bdd6272082 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:54.446513 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdc8f8e-65c8-493f-86eb-ede8bdb587fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:54.447134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdc8f8e-65c8-493f-86eb-ede8bdb587fc None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:54.447417 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdc8f8e-65c8-493f-86eb-ede8bdb587fc None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:54.447693 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdc8f8e-65c8-493f-86eb-ede8bdb587fc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:54.522858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdc8f8e-65c8-493f-86eb-ede8bdb587fc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:54.524118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bcc429-8875-4d5f-90ff-6b01e2d4759d None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:54.524335 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdc8f8e-65c8-493f-86eb-ede8bdb587fc None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:54.524443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bcc429-8875-4d5f-90ff-6b01e2d4759d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:54.524680 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bcc429-8875-4d5f-90ff-6b01e2d4759d None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:54.526281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 493/969] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:54.527104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:54.531131 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3307762-a26f-4377-a803-6e971a818899 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:54.531422 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3307762-a26f-4377-a803-6e971a818899 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:54.531695 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3307762-a26f-4377-a803-6e971a818899 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:54.551154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-e3307762-a26f-4377-a803-6e971a818899 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:54.552267 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e3307762-a26f-4377-a803-6e971a818899 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:54.629456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 494/970] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:54 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:54.639011 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 477/971] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:47:54 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 04 00:47:54.640837 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3307762-a26f-4377-a803-6e971a818899 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:54.689526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:54.758071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3307762-a26f-4377-a803-6e971a818899 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:54.797244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72924ab-7fd4-450f-9e14-0dc718c300f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:54.798407 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72924ab-7fd4-450f-9e14-0dc718c300f0 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:54.798779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72924ab-7fd4-450f-9e14-0dc718c300f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:54.799260 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72924ab-7fd4-450f-9e14-0dc718c300f0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:54.809675 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72924ab-7fd4-450f-9e14-0dc718c300f0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:54.811137 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72924ab-7fd4-450f-9e14-0dc718c300f0 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:54.826991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-a72924ab-7fd4-450f-9e14-0dc718c300f0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:54.826991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:54.826991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:54.826991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:54.826991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:54.826991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:54.826991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:54.826991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:54.826991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:54.826991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:54.826991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:54.826991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:54.826991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:47:54.826991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:47:54.826991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:47:54.826991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:47:54.827921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:54.827921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:54.827921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:54.827921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:54.827921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:54.827921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:54.827921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:54.827921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:54.827921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:54.827921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:54.827921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:54.827921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:54.827921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:54.827921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:54.827921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:54.827921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:54.827921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:47:54.828876 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:47:54.828876 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:47:54.828876 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:54.833130 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 495/972] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:54.834520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdc8f8e-65c8-493f-86eb-ede8bdb587fc None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:54.910836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72924ab-7fd4-450f-9e14-0dc718c300f0 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:54.961712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdc8f8e-65c8-493f-86eb-ede8bdb587fc None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:54.964466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:55.016207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f619f4-5bf4-444f-a147-1572eafb3208 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:55.018576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f619f4-5bf4-444f-a147-1572eafb3208 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:55.018866 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f619f4-5bf4-444f-a147-1572eafb3208 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:55.019141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f619f4-5bf4-444f-a147-1572eafb3208 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:55.019806 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f619f4-5bf4-444f-a147-1572eafb3208 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:55.020117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f619f4-5bf4-444f-a147-1572eafb3208 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:55.021674 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f619f4-5bf4-444f-a147-1572eafb3208 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:55.031314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:55.040120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72924ab-7fd4-450f-9e14-0dc718c300f0 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:55.049377 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 478/973] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:47:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:55.050676 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f619f4-5bf4-444f-a147-1572eafb3208 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:55.091376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7faaa56-d44d-4ddb-b69c-d04ada283e4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:55.092351 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7faaa56-d44d-4ddb-b69c-d04ada283e4e None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:55.092851 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7faaa56-d44d-4ddb-b69c-d04ada283e4e None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:55.093376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7faaa56-d44d-4ddb-b69c-d04ada283e4e None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:55.094328 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7faaa56-d44d-4ddb-b69c-d04ada283e4e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:55.097048 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7faaa56-d44d-4ddb-b69c-d04ada283e4e None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:55.100009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: WARNING keystone.server.flask.common [None req-e7faaa56-d44d-4ddb-b69c-d04ada283e4e None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 04 00:47:55.146253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.sql.core [None req-e7faaa56-d44d-4ddb-b69c-d04ada283e4e None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 04 00:47:55.146253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 04 00:47:55.146253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [parameters: {'user_id': '5d562fce59154487b4d06615c7d463e8', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 04 00:47:55.146253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81558) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 04 00:47:55.167096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-e7faaa56-d44d-4ddb-b69c-d04ada283e4e None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 04 00:47:55.167096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:55.167096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 04 00:47:55.167096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 04 00:47:55.167096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 04 00:47:55.167096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 04 00:47:55.167096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 04 00:47:55.167096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application result = self._query(query) Sep 04 00:47:55.167096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 04 00:47:55.167096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application conn.query(q) Sep 04 00:47:55.167096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 04 00:47:55.167096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 04 00:47:55.167096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 04 00:47:55.167096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application result.read() Sep 04 00:47:55.167096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 04 00:47:55.168038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application next(self.gen) Sep 04 00:47:55.169013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 04 00:47:55.169013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application yield resource Sep 04 00:47:55.169013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 04 00:47:55.169013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application next(self.gen) Sep 04 00:47:55.169013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 04 00:47:55.169013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self.session.rollback() Sep 04 00:47:55.169013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 04 00:47:55.169013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self.force_reraise() Sep 04 00:47:55.169013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 04 00:47:55.169013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise self.value Sep 04 00:47:55.169013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 04 00:47:55.169013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 04 00:47:55.169013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 04 00:47:55.169013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application session.commit() Sep 04 00:47:55.169013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 04 00:47:55.169013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 04 00:47:55.169013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 04 00:47:55.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._prepare_impl() Sep 04 00:47:55.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 04 00:47:55.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self.session.flush() Sep 04 00:47:55.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 04 00:47:55.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._flush(objects) Sep 04 00:47:55.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 04 00:47:55.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 04 00:47:55.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 04 00:47:55.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application compat.raise_( Sep 04 00:47:55.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 04 00:47:55.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception Sep 04 00:47:55.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 04 00:47:55.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application flush_context.execute() Sep 04 00:47:55.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 04 00:47:55.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rec.execute(self) Sep 04 00:47:55.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 04 00:47:55.170556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 04 00:47:55.171946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 04 00:47:55.171946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 04 00:47:55.171946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 04 00:47:55.171946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 04 00:47:55.171946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 04 00:47:55.171946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 04 00:47:55.171946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 04 00:47:55.171946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 04 00:47:55.171946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 04 00:47:55.171946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 04 00:47:55.171946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 04 00:47:55.171946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 04 00:47:55.171946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 04 00:47:55.171946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 04 00:47:55.171946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 04 00:47:55.171946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception Sep 04 00:47:55.174469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 04 00:47:55.174469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 04 00:47:55.174469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 04 00:47:55.174469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 04 00:47:55.174469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 04 00:47:55.174469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application result = self._query(query) Sep 04 00:47:55.174469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 04 00:47:55.174469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application conn.query(q) Sep 04 00:47:55.174469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 04 00:47:55.174469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 04 00:47:55.174469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 04 00:47:55.174469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application result.read() Sep 04 00:47:55.174469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 04 00:47:55.174469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 04 00:47:55.174469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 04 00:47:55.174469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 04 00:47:55.176140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 04 00:47:55.176140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 04 00:47:55.176140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 04 00:47:55.176140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 04 00:47:55.176140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 04 00:47:55.176140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 04 00:47:55.176140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [parameters: {'user_id': '5d562fce59154487b4d06615c7d463e8', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 04 00:47:55.176140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 04 00:47:55.176140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:55.176140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:55.176140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:55.176140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:55.176140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:55.176140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:55.176140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:55.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:55.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:55.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:55.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:55.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:55.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:55.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:55.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 04 00:47:55.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 04 00:47:55.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:55.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:55.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:55.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:55.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:55.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:55.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 04 00:47:55.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 04 00:47:55.186622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 04 00:47:55.186622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 04 00:47:55.186622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 04 00:47:55.186622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 04 00:47:55.186622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 04 00:47:55.186622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:55.186622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 496/974] 10.222.0.42 () {60 vars in 1131 bytes} [Sun Sep 4 00:47:54 2022] POST /identity/v3/users => generated 160 bytes in 258 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:55.186622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7faaa56-d44d-4ddb-b69c-d04ada283e4e None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:55.211261 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f619f4-5bf4-444f-a147-1572eafb3208 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:55.252237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f1f21d-db79-440d-9c2a-87e5d8ffa871 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:55.252883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f1f21d-db79-440d-9c2a-87e5d8ffa871 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:55.253361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f1f21d-db79-440d-9c2a-87e5d8ffa871 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:55.253648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f1f21d-db79-440d-9c2a-87e5d8ffa871 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:55.261470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f1f21d-db79-440d-9c2a-87e5d8ffa871 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:55.262966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f1f21d-db79-440d-9c2a-87e5d8ffa871 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:55.277577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-26f1f21d-db79-440d-9c2a-87e5d8ffa871 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:55.277577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:55.277577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:55.277577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:55.277577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:55.277577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:55.277577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:55.277577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:55.277577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:55.277577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:55.277577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:55.277577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:55.277577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:47:55.277577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:47:55.277577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:47:55.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:47:55.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:55.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:55.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:55.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:55.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:55.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:55.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:55.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:55.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:55.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:55.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:55.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:55.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:55.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:55.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:55.278765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:55.279901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:47:55.279901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:47:55.279901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:47:55.279901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:47:55.279901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:47:55.279901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:47:55.279901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:47:55.279901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:55.279901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 479/975] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:55.287838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f1f21d-db79-440d-9c2a-87e5d8ffa871 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:55.317764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:55.336766 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7faaa56-d44d-4ddb-b69c-d04ada283e4e None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:55.385950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397bf252-16b1-4b50-b874-965c05bea062 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:55.386518 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397bf252-16b1-4b50-b874-965c05bea062 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:55.386798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397bf252-16b1-4b50-b874-965c05bea062 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:55.387185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397bf252-16b1-4b50-b874-965c05bea062 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:55.399976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397bf252-16b1-4b50-b874-965c05bea062 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:55.402356 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f1f21d-db79-440d-9c2a-87e5d8ffa871 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:55.402850 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397bf252-16b1-4b50-b874-965c05bea062 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:55.422763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-397bf252-16b1-4b50-b874-965c05bea062 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 04 00:47:55.422763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:55.422763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:55.422763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:55.422763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:55.422763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:55.422763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:55.422763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:55.422763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:55.422763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:55.422763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:55.422763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:55.422763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:47:55.422763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:47:55.422763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:47:55.422763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:55.424094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:55.425900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:55.425900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:55.425900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:47:55.425900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:47:55.425900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:47:55.425900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:47:55.425900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:47:55.425900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:47:55.425900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 04 00:47:55.425900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:55.425900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 497/976] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:47:55 2022] GET /identity/v3/users/cinder => generated 84 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:55.430847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d63e59-8056-45ba-93c9-148424acff59 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:55.431139 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d63e59-8056-45ba-93c9-148424acff59 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:55.431415 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d63e59-8056-45ba-93c9-148424acff59 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:55.438366 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 498/977] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:55 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:55.440190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d63e59-8056-45ba-93c9-148424acff59 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:55.449924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edafd19-126c-483c-9bb4-503172d0cd3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:55.450537 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edafd19-126c-483c-9bb4-503172d0cd3f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:55.450813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edafd19-126c-483c-9bb4-503172d0cd3f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:55.451082 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edafd19-126c-483c-9bb4-503172d0cd3f None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:55.451825 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edafd19-126c-483c-9bb4-503172d0cd3f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:55.452131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edafd19-126c-483c-9bb4-503172d0cd3f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:55.453588 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edafd19-126c-483c-9bb4-503172d0cd3f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:55.468017 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:55.471137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 480/978] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:47:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:55.474651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b19c003-fe67-4b2f-b3d7-6f32c8eb359c None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:55.474920 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b19c003-fe67-4b2f-b3d7-6f32c8eb359c None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:55.475203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b19c003-fe67-4b2f-b3d7-6f32c8eb359c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:55.494410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-8b19c003-fe67-4b2f-b3d7-6f32c8eb359c None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:55.495939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8b19c003-fe67-4b2f-b3d7-6f32c8eb359c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:55.581746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d63e59-8056-45ba-93c9-148424acff59 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:55.585202 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 481/979] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:55 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:55.587437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b19c003-fe67-4b2f-b3d7-6f32c8eb359c None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:55.629814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:55.630016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052e389f-87cb-4402-a574-6672dce0deb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:55.630698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052e389f-87cb-4402-a574-6672dce0deb5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:55.630983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052e389f-87cb-4402-a574-6672dce0deb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:55.631255 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052e389f-87cb-4402-a574-6672dce0deb5 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:55.631967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052e389f-87cb-4402-a574-6672dce0deb5 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:55.632272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052e389f-87cb-4402-a574-6672dce0deb5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:55.633743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-052e389f-87cb-4402-a574-6672dce0deb5 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:55.683700 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 499/980] 10.222.0.42 () {56 vars in 1101 bytes} [Sun Sep 4 00:47:55 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:55.725350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b19c003-fe67-4b2f-b3d7-6f32c8eb359c None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:55.772688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0ac26-f634-4150-856f-844eb9c2b18d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:55.773342 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0ac26-f634-4150-856f-844eb9c2b18d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:55.773624 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0ac26-f634-4150-856f-844eb9c2b18d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:55.773895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a0ac26-f634-4150-856f-844eb9c2b18d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:55.774694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a0ac26-f634-4150-856f-844eb9c2b18d None admin] RBAC: Adding query filter params (role.id=ace99e88214440f19edb734e93392300, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=70cef7c271e046a3b49230057cc2d28f) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:55.775002 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a0ac26-f634-4150-856f-844eb9c2b18d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:55.776470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a0ac26-f634-4150-856f-844eb9c2b18d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:55.789229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 482/981] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:47:55 2022] GET /identity/v3/role_assignments?user.id=70cef7c271e046a3b49230057cc2d28f&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=ace99e88214440f19edb734e93392300 => generated 608 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:55.902965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052e389f-87cb-4402-a574-6672dce0deb5 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:55.992525 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:56.019732 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052e389f-87cb-4402-a574-6672dce0deb5 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:56.055584 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13859f45-925e-4e2c-9c62-1b34c0e7112d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:56.056220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13859f45-925e-4e2c-9c62-1b34c0e7112d None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:56.056500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13859f45-925e-4e2c-9c62-1b34c0e7112d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:56.056773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13859f45-925e-4e2c-9c62-1b34c0e7112d None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:56.057393 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13859f45-925e-4e2c-9c62-1b34c0e7112d None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:56.058751 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13859f45-925e-4e2c-9c62-1b34c0e7112d None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:56.067514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-13859f45-925e-4e2c-9c62-1b34c0e7112d None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 04 00:47:56.067514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:56.067514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:56.067514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:56.067514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:56.067514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:56.067514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:56.067514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:56.067514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 04 00:47:56.067514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:56.067514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:56.067514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:56.067514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:56.067514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:56.067514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:56.067514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:56.067514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:56.068496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:56.068496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:56.068496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:56.068496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:56.068496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:56.068496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:56.068496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:56.068496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:56.068496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:56.068496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:56.068496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:56.068496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:56.068496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:56.068496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:56.068496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:56.068496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:56.069733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:56.069733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:56.069733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:56.069733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:56.069733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:56.069733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:56.069733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:56.069733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:56.069733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:56.069733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:56.069733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:56.069733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:56.069733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 04 00:47:56.069733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:56.069733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 500/982] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 4 00:47:55 2022] GET /identity/v3/services/object-store => generated 93 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:56.074772 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0ac26-f634-4150-856f-844eb9c2b18d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:56.080058 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf2c3cf-252e-464b-abad-e536dd1d5c93 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:56.080342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf2c3cf-252e-464b-abad-e536dd1d5c93 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:56.080613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf2c3cf-252e-464b-abad-e536dd1d5c93 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:56.082142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 501/983] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:47:56.087476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d48746-e75e-4715-b2e6-aad6b226a33c None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:56.087756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d48746-e75e-4715-b2e6-aad6b226a33c None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:56.088354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d48746-e75e-4715-b2e6-aad6b226a33c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:56.107408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-25d48746-e75e-4715-b2e6-aad6b226a33c None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:56.108922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-25d48746-e75e-4715-b2e6-aad6b226a33c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:56.159643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:56.185112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 502/984] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:56 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:47:56.187534 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a0ac26-f634-4150-856f-844eb9c2b18d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:56.224411 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e56f68e-384d-4b78-882c-3ba51522e408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:56.225017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e56f68e-384d-4b78-882c-3ba51522e408 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:56.225363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e56f68e-384d-4b78-882c-3ba51522e408 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:56.225607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e56f68e-384d-4b78-882c-3ba51522e408 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:56.226241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e56f68e-384d-4b78-882c-3ba51522e408 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:56.226601 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e56f68e-384d-4b78-882c-3ba51522e408 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:56.228043 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e56f68e-384d-4b78-882c-3ba51522e408 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:56.233213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 483/985] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 4 00:47:56 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:56.239589 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d48746-e75e-4715-b2e6-aad6b226a33c None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:56.308005 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:56.340213 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d48746-e75e-4715-b2e6-aad6b226a33c None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:56.376705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8bb2eb-1ce8-43a3-8ccd-f7ca95ddaffe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:56.377336 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8bb2eb-1ce8-43a3-8ccd-f7ca95ddaffe None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:56.377615 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8bb2eb-1ce8-43a3-8ccd-f7ca95ddaffe None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:56.377885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8bb2eb-1ce8-43a3-8ccd-f7ca95ddaffe None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:56.378512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8bb2eb-1ce8-43a3-8ccd-f7ca95ddaffe None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:56.378811 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8bb2eb-1ce8-43a3-8ccd-f7ca95ddaffe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:56.380171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8bb2eb-1ce8-43a3-8ccd-f7ca95ddaffe None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:56.385470 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 503/986] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 4 00:47:56 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:56.391266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e56f68e-384d-4b78-882c-3ba51522e408 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:56.482195 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8bb2eb-1ce8-43a3-8ccd-f7ca95ddaffe None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:56.506899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e56f68e-384d-4b78-882c-3ba51522e408 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:56.543469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a3afd1-41c6-49c5-8864-f0271eb5300c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:56.544068 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a3afd1-41c6-49c5-8864-f0271eb5300c None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:56.544341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a3afd1-41c6-49c5-8864-f0271eb5300c None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:56.544611 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a3afd1-41c6-49c5-8864-f0271eb5300c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:56.545346 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a3afd1-41c6-49c5-8864-f0271eb5300c None admin] RBAC: Adding query filter params (interface=admin, service_id=128f48906f2b459f97dd52bd364518e5, region_id=RegionOne) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:56.545658 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a3afd1-41c6-49c5-8864-f0271eb5300c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:56.546997 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a3afd1-41c6-49c5-8864-f0271eb5300c None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:56.552127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 484/987] 10.222.0.42 () {56 vars in 1249 bytes} [Sun Sep 4 00:47:56 2022] GET /identity/v3/endpoints?service_id=128f48906f2b459f97dd52bd364518e5&interface=admin®ion_id=RegionOne => generated 193 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:56.557706 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a3afd1-41c6-49c5-8864-f0271eb5300c None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:56.567121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:56.593388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8bb2eb-1ce8-43a3-8ccd-f7ca95ddaffe None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:56.622804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:56.628413 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba89ff46-217a-4eae-b6a5-449b4b021d53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:56.629013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba89ff46-217a-4eae-b6a5-449b4b021d53 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:56.629322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba89ff46-217a-4eae-b6a5-449b4b021d53 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:56.629597 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba89ff46-217a-4eae-b6a5-449b4b021d53 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:56.630243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba89ff46-217a-4eae-b6a5-449b4b021d53 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:56.631600 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba89ff46-217a-4eae-b6a5-449b4b021d53 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:56.661018 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a3afd1-41c6-49c5-8864-f0271eb5300c None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:56.695873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8cde13-6a2d-40af-a199-6b2ad4c4bc8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:56.696481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8cde13-6a2d-40af-a199-6b2ad4c4bc8f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:56.696760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8cde13-6a2d-40af-a199-6b2ad4c4bc8f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:56.697031 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8cde13-6a2d-40af-a199-6b2ad4c4bc8f None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:56.697671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8cde13-6a2d-40af-a199-6b2ad4c4bc8f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:56.699025 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8cde13-6a2d-40af-a199-6b2ad4c4bc8f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:56.705574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 485/988] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 4 00:47:56 2022] GET /identity/v3/services => generated 1497 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 00:47:56.717146 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 504/989] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 4 00:47:56 2022] POST /identity/v3/services => generated 242 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 00:47:57.830942 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229891d5-90f7-435d-9929-5eadaef5a6bb None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:57.831513 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229891d5-90f7-435d-9929-5eadaef5a6bb None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:57.831513 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229891d5-90f7-435d-9929-5eadaef5a6bb None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:57.832960 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 486/990] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:57.837972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5373d3ba-4523-4fd7-8635-583590699c56 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:57.838343 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5373d3ba-4523-4fd7-8635-583590699c56 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:57.838521 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5373d3ba-4523-4fd7-8635-583590699c56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:57.856786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-5373d3ba-4523-4fd7-8635-583590699c56 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:57.857995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-5373d3ba-4523-4fd7-8635-583590699c56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:57.932463 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 505/991] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:57 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:58.200487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229891d5-90f7-435d-9929-5eadaef5a6bb None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:58.272383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:58.305059 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229891d5-90f7-435d-9929-5eadaef5a6bb None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:58.340062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3fb5f5-c681-45e3-8865-b838bf3adbad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:58.340697 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3fb5f5-c681-45e3-8865-b838bf3adbad None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:58.340977 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3fb5f5-c681-45e3-8865-b838bf3adbad None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:58.341263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3fb5f5-c681-45e3-8865-b838bf3adbad None admin] PATH_INFO: `/v3/services/image` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:58.341857 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3fb5f5-c681-45e3-8865-b838bf3adbad None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:58.343214 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3fb5f5-c681-45e3-8865-b838bf3adbad None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:58.352353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-5e3fb5f5-c681-45e3-8865-b838bf3adbad None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 04 00:47:58.352353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:58.352353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:58.352353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:58.352353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:58.352353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:58.352353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:58.352353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:58.352353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 04 00:47:58.352353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:58.352353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:58.352353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:58.352353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:58.352353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:58.352353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:58.352353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:58.352353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:58.353410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:58.353410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:58.353410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:58.353410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:58.353410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:58.353410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:58.353410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:58.353410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:58.353410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:58.353410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:58.353410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:58.353410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:58.353410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:58.353410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:58.353410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:58.353410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:58.354233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:58.354233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:58.354233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:58.354233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:58.354233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:58.354233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:58.354233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:58.354233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:58.354233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:58.354233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:58.354233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:58.354233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:58.354233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 04 00:47:58.354233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:58.357458 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 487/992] 10.222.0.42 () {56 vars in 1105 bytes} [Sun Sep 4 00:47:58 2022] GET /identity/v3/services/image => generated 86 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:58.359519 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5373d3ba-4523-4fd7-8635-583590699c56 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:58.467080 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5373d3ba-4523-4fd7-8635-583590699c56 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:58.485829 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:58.504099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47975689-7d01-4b53-ba20-45387171bbf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:58.504710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47975689-7d01-4b53-ba20-45387171bbf7 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:58.505037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47975689-7d01-4b53-ba20-45387171bbf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:58.505345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47975689-7d01-4b53-ba20-45387171bbf7 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:58.505989 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47975689-7d01-4b53-ba20-45387171bbf7 None admin] RBAC: Adding query filter params (name=image) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:58.506331 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47975689-7d01-4b53-ba20-45387171bbf7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:58.507700 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47975689-7d01-4b53-ba20-45387171bbf7 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:58.513570 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 506/993] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:58 2022] GET /identity/v3/services?name=image => generated 122 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:58.519311 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3fb5f5-c681-45e3-8865-b838bf3adbad None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:58.621968 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3fb5f5-c681-45e3-8865-b838bf3adbad None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:58.656324 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d04dfba-6f49-43a4-9495-520e4b8bc613 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:58.656933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d04dfba-6f49-43a4-9495-520e4b8bc613 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:58.657866 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d04dfba-6f49-43a4-9495-520e4b8bc613 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:58.658142 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d04dfba-6f49-43a4-9495-520e4b8bc613 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:58.658788 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d04dfba-6f49-43a4-9495-520e4b8bc613 None admin] RBAC: Adding query filter params (type=image) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:58.659088 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d04dfba-6f49-43a4-9495-520e4b8bc613 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:58.660448 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d04dfba-6f49-43a4-9495-520e4b8bc613 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:58.661779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:58.666284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 488/994] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:47:58 2022] GET /identity/v3/services?type=image => generated 122 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:58.672923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee15ba7-c064-49bf-a948-fd69eb1d1b28 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:58.673258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee15ba7-c064-49bf-a948-fd69eb1d1b28 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:58.673524 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee15ba7-c064-49bf-a948-fd69eb1d1b28 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:58.675092 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 507/995] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:47:58.680700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ddbe16-e70c-41a0-84c8-755343521dbc None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:58.681000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ddbe16-e70c-41a0-84c8-755343521dbc None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:58.681305 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ddbe16-e70c-41a0-84c8-755343521dbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:58.700030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78aa83ad-1828-4df5-a7fd-49de6dd1db6b None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:58.700324 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78aa83ad-1828-4df5-a7fd-49de6dd1db6b None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:58.700609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78aa83ad-1828-4df5-a7fd-49de6dd1db6b None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:58.701696 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-35ddbe16-e70c-41a0-84c8-755343521dbc None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:58.702358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 508/996] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:47:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:47:58.703309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-35ddbe16-e70c-41a0-84c8-755343521dbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:58.708206 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aef60d-2f5c-441a-ad43-73cb3a482111 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:58.708488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aef60d-2f5c-441a-ad43-73cb3a482111 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:58.708772 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aef60d-2f5c-441a-ad43-73cb3a482111 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:58.728968 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-d4aef60d-2f5c-441a-ad43-73cb3a482111 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:47:58.730215 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d4aef60d-2f5c-441a-ad43-73cb3a482111 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:47:58.784494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 489/997] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:58 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:47:58.812488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 509/998] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:47:58 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:47:59.057879 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ddbe16-e70c-41a0-84c8-755343521dbc None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:59.083887 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4aef60d-2f5c-441a-ad43-73cb3a482111 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:59.169020 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ddbe16-e70c-41a0-84c8-755343521dbc None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:59.193246 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4aef60d-2f5c-441a-ad43-73cb3a482111 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:59.199631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:59.208749 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:59.211686 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a69f27-4dc8-468f-85bc-5a6263707048 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:59.212760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a69f27-4dc8-468f-85bc-5a6263707048 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:59.213572 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a69f27-4dc8-468f-85bc-5a6263707048 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:59.213990 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a69f27-4dc8-468f-85bc-5a6263707048 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:59.221022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a69f27-4dc8-468f-85bc-5a6263707048 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:59.221022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a69f27-4dc8-468f-85bc-5a6263707048 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:59.228275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-06a69f27-4dc8-468f-85bc-5a6263707048 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 04 00:47:59.228275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:59.228275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:59.228275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:59.228275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:59.228275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:59.228275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:59.228275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:59.228275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 04 00:47:59.228275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:59.228275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:59.228275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:59.228275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:59.228275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:59.228275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:59.228275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:59.228275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:59.229427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:59.229427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:59.229427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:59.229427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:59.229427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:59.229427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:59.229427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:59.229427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:59.229427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:59.229427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:59.229427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:59.229427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:59.229427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:59.229427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:59.229427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:59.229427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:59.230469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:59.230469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:59.230469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:59.230469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:59.230469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:59.230469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:59.230469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:59.230469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:59.230469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:59.230469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:59.230469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:59.230469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:59.230469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 04 00:47:59.230469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:47:59.231374 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebc5c68-228f-42ac-b85a-edf1f7f848f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:59.231374 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebc5c68-228f-42ac-b85a-edf1f7f848f4 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:59.231374 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebc5c68-228f-42ac-b85a-edf1f7f848f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:59.231638 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebc5c68-228f-42ac-b85a-edf1f7f848f4 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:59.232090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebc5c68-228f-42ac-b85a-edf1f7f848f4 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:59.233367 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 490/999] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:47:59 2022] GET /identity/v3/services/compute => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:47:59.233556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebc5c68-228f-42ac-b85a-edf1f7f848f4 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:59.239473 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a69f27-4dc8-468f-85bc-5a6263707048 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:59.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-4ebc5c68-228f-42ac-b85a-edf1f7f848f4 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 04 00:47:59.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:59.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:47:59.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:47:59.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:47:59.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:47:59.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:47:59.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:59.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 04 00:47:59.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:59.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:47:59.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:59.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:47:59.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:47:59.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:47:59.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:47:59.243389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:47:59.244721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:47:59.244721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:47:59.244721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:47:59.244721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:47:59.244721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:47:59.244721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:47:59.244721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:47:59.244721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:47:59.244721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:47:59.244721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:47:59.244721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:47:59.244721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:47:59.244721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:47:59.244721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:47:59.244721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:47:59.244721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:47:59.245619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:47:59.245619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:47:59.245619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:47:59.245619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:47:59.245619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:47:59.245619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:47:59.245619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:47:59.245619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:47:59.245619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:47:59.245619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:47:59.245619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:47:59.245619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:47:59.245619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 04 00:47:59.245619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:47:59.245619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 510/1000] 10.222.0.42 () {56 vars in 1126 bytes} [Sun Sep 4 00:47:59 2022] GET /identity/v3/services/object-store => generated 93 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:47:59.252850 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebc5c68-228f-42ac-b85a-edf1f7f848f4 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:59.346184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a69f27-4dc8-468f-85bc-5a6263707048 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:59.365178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebc5c68-228f-42ac-b85a-edf1f7f848f4 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:59.382822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133eabcd-dbeb-4236-95a3-7649497a1fc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:59.383589 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133eabcd-dbeb-4236-95a3-7649497a1fc9 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:59.383876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133eabcd-dbeb-4236-95a3-7649497a1fc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:59.384149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133eabcd-dbeb-4236-95a3-7649497a1fc9 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:59.384779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133eabcd-dbeb-4236-95a3-7649497a1fc9 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:59.385101 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133eabcd-dbeb-4236-95a3-7649497a1fc9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:59.386478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133eabcd-dbeb-4236-95a3-7649497a1fc9 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:59.393121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 491/1001] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 4 00:47:59 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:59.397146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133eabcd-dbeb-4236-95a3-7649497a1fc9 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:59.408465 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121efead-cc51-4e2d-9e94-f66b7909a0b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:59.409065 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121efead-cc51-4e2d-9e94-f66b7909a0b4 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:59.409378 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121efead-cc51-4e2d-9e94-f66b7909a0b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:59.409656 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121efead-cc51-4e2d-9e94-f66b7909a0b4 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:59.410425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121efead-cc51-4e2d-9e94-f66b7909a0b4 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:59.410599 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121efead-cc51-4e2d-9e94-f66b7909a0b4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:59.411961 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121efead-cc51-4e2d-9e94-f66b7909a0b4 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:59.421545 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 511/1002] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 4 00:47:59 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:59.422640 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121efead-cc51-4e2d-9e94-f66b7909a0b4 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:59.456964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:59.474635 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:59.503638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133eabcd-dbeb-4236-95a3-7649497a1fc9 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:59.535803 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121efead-cc51-4e2d-9e94-f66b7909a0b4 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:59.538102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494c3ca3-269f-4c0f-beaa-4ec21c10e643 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:59.538699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494c3ca3-269f-4c0f-beaa-4ec21c10e643 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:59.538980 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494c3ca3-269f-4c0f-beaa-4ec21c10e643 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:59.539248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494c3ca3-269f-4c0f-beaa-4ec21c10e643 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:59.539875 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494c3ca3-269f-4c0f-beaa-4ec21c10e643 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:59.540177 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494c3ca3-269f-4c0f-beaa-4ec21c10e643 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:59.541558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494c3ca3-269f-4c0f-beaa-4ec21c10e643 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:59.552077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 492/1003] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 4 00:47:59 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:59.552944 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494c3ca3-269f-4c0f-beaa-4ec21c10e643 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:59.573577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bdd637-3f79-48d5-95cc-323780f09d28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:59.574598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bdd637-3f79-48d5-95cc-323780f09d28 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:59.574884 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bdd637-3f79-48d5-95cc-323780f09d28 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:59.575156 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bdd637-3f79-48d5-95cc-323780f09d28 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:59.575954 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bdd637-3f79-48d5-95cc-323780f09d28 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:59.576256 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bdd637-3f79-48d5-95cc-323780f09d28 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:59.577646 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bdd637-3f79-48d5-95cc-323780f09d28 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:59.583379 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 512/1004] 10.222.0.42 () {56 vars in 1122 bytes} [Sun Sep 4 00:47:59 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:47:59.588800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bdd637-3f79-48d5-95cc-323780f09d28 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:59.660868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494c3ca3-269f-4c0f-beaa-4ec21c10e643 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:59.681494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:59.696436 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bdd637-3f79-48d5-95cc-323780f09d28 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:59.697433 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9d1f52-3f40-43d5-835d-beab97a3a238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:59.698040 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9d1f52-3f40-43d5-835d-beab97a3a238 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:59.698320 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9d1f52-3f40-43d5-835d-beab97a3a238 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:59.698595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9d1f52-3f40-43d5-835d-beab97a3a238 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:59.699291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9d1f52-3f40-43d5-835d-beab97a3a238 None admin] RBAC: Adding query filter params (interface=public, service_id=e7c5c78bb351436fae9af907894d71bb, region_id=RegionOne) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:47:59.699509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:47:59.699624 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9d1f52-3f40-43d5-835d-beab97a3a238 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:59.701135 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9d1f52-3f40-43d5-835d-beab97a3a238 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:59.707294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 493/1005] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Sep 4 00:47:59 2022] GET /identity/v3/endpoints?service_id=e7c5c78bb351436fae9af907894d71bb&interface=public®ion_id=RegionOne => generated 194 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:47:59.712623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9d1f52-3f40-43d5-835d-beab97a3a238 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:47:59.734478 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d056544-5f3e-4d3c-b5a6-8fa9784cdb64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:59.735116 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d056544-5f3e-4d3c-b5a6-8fa9784cdb64 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:59.735676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d056544-5f3e-4d3c-b5a6-8fa9784cdb64 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:59.735972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d056544-5f3e-4d3c-b5a6-8fa9784cdb64 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:59.736633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d056544-5f3e-4d3c-b5a6-8fa9784cdb64 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:59.738244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d056544-5f3e-4d3c-b5a6-8fa9784cdb64 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:59.814274 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9d1f52-3f40-43d5-835d-beab97a3a238 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:47:59.822573 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 513/1006] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 00:47:59 2022] POST /identity/v3/endpoints => generated 325 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 00:47:59.857315 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c598fa84-be1e-4304-b71c-78848786252d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:47:59.857937 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c598fa84-be1e-4304-b71c-78848786252d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:47:59.858218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c598fa84-be1e-4304-b71c-78848786252d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:47:59.858484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c598fa84-be1e-4304-b71c-78848786252d None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:47:59.859098 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c598fa84-be1e-4304-b71c-78848786252d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:47:59.860503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c598fa84-be1e-4304-b71c-78848786252d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:47:59.877110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 494/1007] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 4 00:47:59 2022] GET /identity/v3/services => generated 1727 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 00:48:00.597480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad54e22f-d678-4b4c-9813-cfa5215bd33e None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:00.598021 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad54e22f-d678-4b4c-9813-cfa5215bd33e None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:00.598021 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad54e22f-d678-4b4c-9813-cfa5215bd33e None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:00.599504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 514/1008] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:00.604334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd854ada-ceea-4d91-883d-45c7f50f2c52 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:00.604707 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd854ada-ceea-4d91-883d-45c7f50f2c52 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:00.604892 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd854ada-ceea-4d91-883d-45c7f50f2c52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:00.622683 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-cd854ada-ceea-4d91-883d-45c7f50f2c52 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:00.624124 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-cd854ada-ceea-4d91-883d-45c7f50f2c52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:00.666324 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:00.692734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 495/1009] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:00 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:00.960289 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad54e22f-d678-4b4c-9813-cfa5215bd33e None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:01.064384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad54e22f-d678-4b4c-9813-cfa5215bd33e None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:01.098801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71217ac6-4dc7-459e-a8a2-c9059973e463 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:01.099402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71217ac6-4dc7-459e-a8a2-c9059973e463 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:01.099681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71217ac6-4dc7-459e-a8a2-c9059973e463 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:01.099948 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71217ac6-4dc7-459e-a8a2-c9059973e463 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:01.100657 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71217ac6-4dc7-459e-a8a2-c9059973e463 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:01.102114 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71217ac6-4dc7-459e-a8a2-c9059973e463 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:01.210653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 515/1010] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 4 00:48:00 2022] POST /identity/v3/services => generated 242 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:48:01.760623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc57a72-e70b-49aa-863c-414717c12569 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:01.761329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc57a72-e70b-49aa-863c-414717c12569 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:01.761329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc57a72-e70b-49aa-863c-414717c12569 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:01.762586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 496/1011] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:01.768468 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f95ccb-e243-4dd1-8937-c8ea1135bb35 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:01.768870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f95ccb-e243-4dd1-8937-c8ea1135bb35 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:01.769038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f95ccb-e243-4dd1-8937-c8ea1135bb35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:01.787621 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-f6f95ccb-e243-4dd1-8937-c8ea1135bb35 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:01.788798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-f6f95ccb-e243-4dd1-8937-c8ea1135bb35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:01.806983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:01.817166 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f417ed0-8795-46b4-abf3-18cba6f2f3a0 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:01.817455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f417ed0-8795-46b4-abf3-18cba6f2f3a0 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:01.817731 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f417ed0-8795-46b4-abf3-18cba6f2f3a0 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:01.821117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 497/1012] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:48:01.824204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ad5a0a-5780-401f-97bd-bbe28bbcff91 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:01.825242 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ad5a0a-5780-401f-97bd-bbe28bbcff91 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:01.825242 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ad5a0a-5780-401f-97bd-bbe28bbcff91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:01.843211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-58ad5a0a-5780-401f-97bd-bbe28bbcff91 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:01.844403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-58ad5a0a-5780-401f-97bd-bbe28bbcff91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:01.864807 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 516/1013] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:01 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:01.918562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 498/1014] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:01 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:48:02.131601 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f95ccb-e243-4dd1-8937-c8ea1135bb35 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:02.191181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ad5a0a-5780-401f-97bd-bbe28bbcff91 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:02.238462 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f95ccb-e243-4dd1-8937-c8ea1135bb35 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:02.276573 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824317a6-906b-4c5f-8e48-76a3829f50db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:02.277230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824317a6-906b-4c5f-8e48-76a3829f50db None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:02.277509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824317a6-906b-4c5f-8e48-76a3829f50db None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:02.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824317a6-906b-4c5f-8e48-76a3829f50db None admin] PATH_INFO: `/v3/domains/default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:02.278875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824317a6-906b-4c5f-8e48-76a3829f50db None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:02.280305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824317a6-906b-4c5f-8e48-76a3829f50db None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:02.282140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 517/1015] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:48:02 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:02.287867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824317a6-906b-4c5f-8e48-76a3829f50db None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:02.300222 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ad5a0a-5780-401f-97bd-bbe28bbcff91 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:02.302599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:02.309192 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:02.338025 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be663661-bb5b-4365-bde9-288706c75ea4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:02.338847 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be663661-bb5b-4365-bde9-288706c75ea4 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:02.339235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be663661-bb5b-4365-bde9-288706c75ea4 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:02.339409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be663661-bb5b-4365-bde9-288706c75ea4 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:02.340008 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be663661-bb5b-4365-bde9-288706c75ea4 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:02.341404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be663661-bb5b-4365-bde9-288706c75ea4 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:02.350550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-be663661-bb5b-4365-bde9-288706c75ea4 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 04 00:48:02.350550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:02.350550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:48:02.350550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:48:02.350550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:48:02.350550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:48:02.350550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:48:02.350550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:48:02.350550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 04 00:48:02.350550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:02.350550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:48:02.350550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:02.350550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:02.350550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:02.350550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:02.350550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:02.350550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:02.351556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:02.351556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:02.351556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:02.351556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:02.351556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:02.351556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:02.351556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:48:02.351556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:48:02.351556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:48:02.351556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:48:02.351556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:02.351556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:02.351556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:02.351556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:02.351556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:02.351556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:02.352393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:02.352393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:02.352393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:02.352393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:02.352393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:02.352393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:02.352393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:02.352393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:02.352393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:02.352393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:02.352393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:48:02.352393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:48:02.352393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 04 00:48:02.352393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:02.352393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 499/1016] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:48:02 2022] GET /identity/v3/services/compute => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:48:02.357462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be663661-bb5b-4365-bde9-288706c75ea4 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:02.400120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824317a6-906b-4c5f-8e48-76a3829f50db None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:02.437173 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a6aca7-7640-46d1-b85f-b3cd75053298 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:02.438205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a6aca7-7640-46d1-b85f-b3cd75053298 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:02.438205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a6aca7-7640-46d1-b85f-b3cd75053298 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:02.439153 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a6aca7-7640-46d1-b85f-b3cd75053298 None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:02.439860 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a6aca7-7640-46d1-b85f-b3cd75053298 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:02.441310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a6aca7-7640-46d1-b85f-b3cd75053298 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:02.467407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be663661-bb5b-4365-bde9-288706c75ea4 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:02.502001 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e13a2e1-27e6-4e46-bca7-d737dbe7015a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:02.502625 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e13a2e1-27e6-4e46-bca7-d737dbe7015a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:02.502906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e13a2e1-27e6-4e46-bca7-d737dbe7015a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:02.503175 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e13a2e1-27e6-4e46-bca7-d737dbe7015a None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:02.503810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e13a2e1-27e6-4e46-bca7-d737dbe7015a None admin] RBAC: Adding query filter params (name=compute) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:02.504115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e13a2e1-27e6-4e46-bca7-d737dbe7015a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:02.505599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e13a2e1-27e6-4e46-bca7-d737dbe7015a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:02.512465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 500/1017] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 4 00:48:02 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:02.518782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e13a2e1-27e6-4e46-bca7-d737dbe7015a None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:02.531607 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 518/1018] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 4 00:48:02 2022] POST /identity/v3/projects => generated 311 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:48:02.563324 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:02.645474 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e13a2e1-27e6-4e46-bca7-d737dbe7015a None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:02.683100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1206ca2-2bb1-4de5-8b54-8cdef110135f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:02.683762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1206ca2-2bb1-4de5-8b54-8cdef110135f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:02.684046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1206ca2-2bb1-4de5-8b54-8cdef110135f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:02.684318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1206ca2-2bb1-4de5-8b54-8cdef110135f None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:02.684945 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1206ca2-2bb1-4de5-8b54-8cdef110135f None admin] RBAC: Adding query filter params (type=compute) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:02.685289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1206ca2-2bb1-4de5-8b54-8cdef110135f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:02.686647 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1206ca2-2bb1-4de5-8b54-8cdef110135f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:02.692555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 501/1019] 10.222.0.42 () {56 vars in 1112 bytes} [Sun Sep 4 00:48:02 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:02.698249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a6aca7-7640-46d1-b85f-b3cd75053298 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:02.774190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:02.805376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a6aca7-7640-46d1-b85f-b3cd75053298 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:02.840175 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9d6dbb-8b63-4a82-b5f1-1c2a68dd9d4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:02.840785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9d6dbb-8b63-4a82-b5f1-1c2a68dd9d4b None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:02.841061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9d6dbb-8b63-4a82-b5f1-1c2a68dd9d4b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:02.841352 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9d6dbb-8b63-4a82-b5f1-1c2a68dd9d4b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:02.842024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9d6dbb-8b63-4a82-b5f1-1c2a68dd9d4b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:02.843441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9d6dbb-8b63-4a82-b5f1-1c2a68dd9d4b None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:02.921125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 519/1020] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 00:48:02 2022] POST /identity/v3/endpoints => generated 334 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 00:48:03.222128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69dc875-1a0d-4bef-96b7-5acd78784b02 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:03.222769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69dc875-1a0d-4bef-96b7-5acd78784b02 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:03.222769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69dc875-1a0d-4bef-96b7-5acd78784b02 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:03.224243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 502/1021] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:03.230461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e68cf21-4661-4808-8cdf-45819950a926 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:03.230751 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e68cf21-4661-4808-8cdf-45819950a926 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:03.231025 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e68cf21-4661-4808-8cdf-45819950a926 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:03.251622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-5e68cf21-4661-4808-8cdf-45819950a926 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:03.252831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-5e68cf21-4661-4808-8cdf-45819950a926 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:03.328579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 520/1022] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:03 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:03.605104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69dc875-1a0d-4bef-96b7-5acd78784b02 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:03.705236 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:03.712420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69dc875-1a0d-4bef-96b7-5acd78784b02 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:03.745936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd579021-54b2-4fb0-836d-567fa051dc94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:03.746560 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd579021-54b2-4fb0-836d-567fa051dc94 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:03.746839 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd579021-54b2-4fb0-836d-567fa051dc94 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:03.747107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd579021-54b2-4fb0-836d-567fa051dc94 None admin] PATH_INFO: `/v3/services/image` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:03.747688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd579021-54b2-4fb0-836d-567fa051dc94 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:03.749312 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd579021-54b2-4fb0-836d-567fa051dc94 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:03.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-dd579021-54b2-4fb0-836d-567fa051dc94 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 04 00:48:03.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:03.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:48:03.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:48:03.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:48:03.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:48:03.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:48:03.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:48:03.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 04 00:48:03.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:03.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:48:03.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:03.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:03.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:03.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:03.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:03.758778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:03.759780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:03.759780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:03.759780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:03.759780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:03.759780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:03.759780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:03.759780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:48:03.759780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:48:03.759780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:48:03.759780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:48:03.759780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:03.759780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:03.759780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:03.759780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:03.759780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:03.759780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:03.760630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:03.760630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:03.760630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:03.760630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:03.760630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:03.760630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:03.760630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:03.760630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:03.760630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:03.760630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:03.760630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:48:03.760630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:48:03.760630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 04 00:48:03.760630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:03.762181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 503/1023] 10.222.0.42 () {56 vars in 1105 bytes} [Sun Sep 4 00:48:03 2022] GET /identity/v3/services/image => generated 86 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:48:03.765777 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e68cf21-4661-4808-8cdf-45819950a926 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:03.817276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:03.864035 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e68cf21-4661-4808-8cdf-45819950a926 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:03.899807 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a2146c-1d03-4d21-afbd-4862673b0718 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:03.900717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a2146c-1d03-4d21-afbd-4862673b0718 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:03.901230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a2146c-1d03-4d21-afbd-4862673b0718 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:03.901745 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a2146c-1d03-4d21-afbd-4862673b0718 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:03.902593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a2146c-1d03-4d21-afbd-4862673b0718 None admin] RBAC: Adding query filter params (name=image) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:03.903092 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a2146c-1d03-4d21-afbd-4862673b0718 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:03.904712 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a2146c-1d03-4d21-afbd-4862673b0718 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:03.909868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 521/1024] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:48:03 2022] GET /identity/v3/services?name=image => generated 122 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:03.915787 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd579021-54b2-4fb0-836d-567fa051dc94 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:04.020091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd579021-54b2-4fb0-836d-567fa051dc94 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:04.054303 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01bd6bc-a8aa-418a-977f-f4925fb2bfc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:04.054904 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01bd6bc-a8aa-418a-977f-f4925fb2bfc2 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:04.055181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01bd6bc-a8aa-418a-977f-f4925fb2bfc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:04.055451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01bd6bc-a8aa-418a-977f-f4925fb2bfc2 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:04.056111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01bd6bc-a8aa-418a-977f-f4925fb2bfc2 None admin] RBAC: Adding query filter params (type=image) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:04.056413 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01bd6bc-a8aa-418a-977f-f4925fb2bfc2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:04.057783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01bd6bc-a8aa-418a-977f-f4925fb2bfc2 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:04.062636 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 504/1025] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:48:03 2022] GET /identity/v3/services?type=image => generated 350 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:04.068786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a2146c-1d03-4d21-afbd-4862673b0718 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:04.172539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a2146c-1d03-4d21-afbd-4862673b0718 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:04.188848 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:04.206952 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f50c9b8-8edd-48c9-98b3-78ab150a5d45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:04.207564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f50c9b8-8edd-48c9-98b3-78ab150a5d45 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:04.207838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f50c9b8-8edd-48c9-98b3-78ab150a5d45 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:04.208107 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f50c9b8-8edd-48c9-98b3-78ab150a5d45 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:04.208799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f50c9b8-8edd-48c9-98b3-78ab150a5d45 None admin] RBAC: Adding query filter params (interface=public, service_id=af36b8f7cc674adb909a99df2b90963a, region_id=RegionOne) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:04.209125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f50c9b8-8edd-48c9-98b3-78ab150a5d45 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:04.210484 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f50c9b8-8edd-48c9-98b3-78ab150a5d45 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:04.220950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 522/1026] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Sep 4 00:48:04 2022] GET /identity/v3/endpoints?service_id=af36b8f7cc674adb909a99df2b90963a&interface=public®ion_id=RegionOne => generated 194 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:04.222510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01bd6bc-a8aa-418a-977f-f4925fb2bfc2 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:04.248966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:04.326386 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01bd6bc-a8aa-418a-977f-f4925fb2bfc2 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:04.361334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ffc98d-59e5-4969-82c7-93469a1f0ab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:04.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ffc98d-59e5-4969-82c7-93469a1f0ab4 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:04.362231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ffc98d-59e5-4969-82c7-93469a1f0ab4 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:04.362487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ffc98d-59e5-4969-82c7-93469a1f0ab4 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:04.363094 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ffc98d-59e5-4969-82c7-93469a1f0ab4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:04.364440 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ffc98d-59e5-4969-82c7-93469a1f0ab4 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:04.371338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 505/1027] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 4 00:48:04 2022] GET /identity/v3/services => generated 1957 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 00:48:04.573622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9e5183-4cbf-478c-8a60-880304786ff2 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:04.573946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9e5183-4cbf-478c-8a60-880304786ff2 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:04.574183 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d9e5183-4cbf-478c-8a60-880304786ff2 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:04.575531 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 523/1028] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:04.581565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e939e9-793b-489f-9ee0-1b5a8c2e5d63 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:04.581855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e939e9-793b-489f-9ee0-1b5a8c2e5d63 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:04.582131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e939e9-793b-489f-9ee0-1b5a8c2e5d63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:04.601010 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-f0e939e9-793b-489f-9ee0-1b5a8c2e5d63 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:04.602232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f0e939e9-793b-489f-9ee0-1b5a8c2e5d63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:04.693113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 506/1029] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:04 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:04.960147 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9e5183-4cbf-478c-8a60-880304786ff2 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:04.984573 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065c0149-7875-4326-804f-0eb4777ba1ec None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:04.984919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065c0149-7875-4326-804f-0eb4777ba1ec None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:04.985783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065c0149-7875-4326-804f-0eb4777ba1ec None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:04.987147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 507/1030] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:48:04.992297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e013719e-d8e5-44fe-9bfc-219973171bc5 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:04.992581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e013719e-d8e5-44fe-9bfc-219973171bc5 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:04.992860 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e013719e-d8e5-44fe-9bfc-219973171bc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:05.012869 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-e013719e-d8e5-44fe-9bfc-219973171bc5 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:05.014076 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e013719e-d8e5-44fe-9bfc-219973171bc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:05.028983 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:05.078049 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d9e5183-4cbf-478c-8a60-880304786ff2 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:05.087119 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 508/1031] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:04 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:48:05.114168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd987ed2-0da4-475c-a699-8cd3aa68f119 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:05.114801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd987ed2-0da4-475c-a699-8cd3aa68f119 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:05.115082 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd987ed2-0da4-475c-a699-8cd3aa68f119 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:05.115355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd987ed2-0da4-475c-a699-8cd3aa68f119 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:05.116478 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd987ed2-0da4-475c-a699-8cd3aa68f119 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:05.118019 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd987ed2-0da4-475c-a699-8cd3aa68f119 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:05.119725 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 524/1032] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:48:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:05.125526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e013719e-d8e5-44fe-9bfc-219973171bc5 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:05.225925 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e013719e-d8e5-44fe-9bfc-219973171bc5 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:05.259590 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1e969e-97eb-437c-a293-3fed143a34a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:05.260192 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1e969e-97eb-437c-a293-3fed143a34a1 None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:05.260468 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1e969e-97eb-437c-a293-3fed143a34a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:05.260737 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1e969e-97eb-437c-a293-3fed143a34a1 None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:05.261427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1e969e-97eb-437c-a293-3fed143a34a1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:05.262836 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1e969e-97eb-437c-a293-3fed143a34a1 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:05.341450 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 509/1033] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:48:05 2022] POST /identity/v3/users => generated 285 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:48:05.368670 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd987ed2-0da4-475c-a699-8cd3aa68f119 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:05.430374 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:05.485991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd987ed2-0da4-475c-a699-8cd3aa68f119 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:05.524785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cab753d-b45a-4039-9f54-39bdc040c1f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:05.525445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab753d-b45a-4039-9f54-39bdc040c1f0 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:05.525727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab753d-b45a-4039-9f54-39bdc040c1f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:05.526000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cab753d-b45a-4039-9f54-39bdc040c1f0 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:05.536342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cab753d-b45a-4039-9f54-39bdc040c1f0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:48:05.541174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-0cab753d-b45a-4039-9f54-39bdc040c1f0 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 04 00:48:05.541174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:05.541174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:05.541174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:05.541174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:05.541174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:05.541174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:05.541174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:05.541174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:05.541174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:05.541174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:05.541174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:05.541174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:48:05.541174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:48:05.541174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:48:05.541174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:48:05.542099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:48:05.542099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:48:05.542099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:48:05.542099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:48:05.542099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:05.542099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:05.542099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:05.542099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:05.542099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:05.542099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:05.542099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:05.542099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:05.542099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:05.542099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:05.542099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:05.542099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:05.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:05.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:05.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:05.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:05.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:48:05.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:48:05.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:48:05.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:48:05.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:48:05.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:48:05.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 04 00:48:05.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:48:05.542929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 525/1034] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Sep 4 00:48:05 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:48:05.549106 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1e969e-97eb-437c-a293-3fed143a34a1 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:05.553060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:05.658457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1e969e-97eb-437c-a293-3fed143a34a1 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:05.693519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2ba609-ceba-4cef-8b80-7f8bfb6ca38f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:05.694130 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ba609-ceba-4cef-8b80-7f8bfb6ca38f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:05.694411 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ba609-ceba-4cef-8b80-7f8bfb6ca38f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:05.694683 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2ba609-ceba-4cef-8b80-7f8bfb6ca38f None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:05.695355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2ba609-ceba-4cef-8b80-7f8bfb6ca38f None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:05.695659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2ba609-ceba-4cef-8b80-7f8bfb6ca38f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:05.697028 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2ba609-ceba-4cef-8b80-7f8bfb6ca38f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:05.705574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 510/1035] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Sep 4 00:48:05 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:05.711006 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cab753d-b45a-4039-9f54-39bdc040c1f0 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:05.812902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cab753d-b45a-4039-9f54-39bdc040c1f0 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:05.823092 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:05.848578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee432bd-996b-4f11-af41-4315172eb004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:05.849250 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee432bd-996b-4f11-af41-4315172eb004 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:05.849534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee432bd-996b-4f11-af41-4315172eb004 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:05.849812 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee432bd-996b-4f11-af41-4315172eb004 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:05.860080 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee432bd-996b-4f11-af41-4315172eb004 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:05.861705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee432bd-996b-4f11-af41-4315172eb004 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:05.877944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-4ee432bd-996b-4f11-af41-4315172eb004 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:48:05.877944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:05.877944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:05.877944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:05.877944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:05.877944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:05.877944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:05.877944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:05.877944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:05.877944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:05.877944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:05.877944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:05.877944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:48:05.877944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:48:05.877944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:48:05.877944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:48:05.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:05.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:05.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:05.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:05.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:05.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:05.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:05.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:05.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:05.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:05.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:05.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:05.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:05.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:05.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:05.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:05.878934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:48:05.879838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:48:05.879838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:48:05.879838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:48:05.881155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 526/1036] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:48:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:48:05.885155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2ba609-ceba-4cef-8b80-7f8bfb6ca38f None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:05.952180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:05.989086 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2ba609-ceba-4cef-8b80-7f8bfb6ca38f None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:06.022583 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed670328-c97d-4ef5-a323-88225c797ccc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:06.023185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed670328-c97d-4ef5-a323-88225c797ccc None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:06.023460 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed670328-c97d-4ef5-a323-88225c797ccc None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:06.023729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed670328-c97d-4ef5-a323-88225c797ccc None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:06.024367 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed670328-c97d-4ef5-a323-88225c797ccc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:06.024669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed670328-c97d-4ef5-a323-88225c797ccc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:06.026039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed670328-c97d-4ef5-a323-88225c797ccc None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:06.038411 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 511/1037] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:48:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:06.043804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee432bd-996b-4f11-af41-4315172eb004 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:06.144255 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee432bd-996b-4f11-af41-4315172eb004 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:06.177415 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850c27ea-199e-44d7-822b-33082f7b2b8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:06.178043 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850c27ea-199e-44d7-822b-33082f7b2b8d None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:06.178317 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850c27ea-199e-44d7-822b-33082f7b2b8d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:06.178586 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850c27ea-199e-44d7-822b-33082f7b2b8d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:06.186713 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850c27ea-199e-44d7-822b-33082f7b2b8d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:06.188193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850c27ea-199e-44d7-822b-33082f7b2b8d None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:06.199168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-850c27ea-199e-44d7-822b-33082f7b2b8d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:48:06.199168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:06.199168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:06.199168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:06.199168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:06.199168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:06.199168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:06.199168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:06.199168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:06.199168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:06.199168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:06.199168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:06.199168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:48:06.199168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:48:06.199168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:48:06.199168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:06.200091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:06.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:06.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:06.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:48:06.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:48:06.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:48:06.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:48:06.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:48:06.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:48:06.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:48:06.201046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:48:06.202220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 527/1038] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 4 00:48:06 2022] GET /identity/v3/users/nova => generated 82 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:48:06.206816 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed670328-c97d-4ef5-a323-88225c797ccc None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:06.310515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed670328-c97d-4ef5-a323-88225c797ccc None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:06.321512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79454299-ac7f-40dc-ab7a-0c2f8d4ec683 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:06.321859 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79454299-ac7f-40dc-ab7a-0c2f8d4ec683 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:06.322071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79454299-ac7f-40dc-ab7a-0c2f8d4ec683 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:06.323396 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 528/1039] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:48:06.328109 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab28717-921f-4bb0-91e8-815b5e399bd2 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:06.328392 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab28717-921f-4bb0-91e8-815b5e399bd2 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:06.328785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab28717-921f-4bb0-91e8-815b5e399bd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:06.337870 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:06.347105 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c187bf51-c5c7-4677-ad59-8c5dc9fc761b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:06.353186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c187bf51-c5c7-4677-ad59-8c5dc9fc761b None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:06.353186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c187bf51-c5c7-4677-ad59-8c5dc9fc761b None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:06.353186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c187bf51-c5c7-4677-ad59-8c5dc9fc761b None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:06.353411 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-fab28717-921f-4bb0-91e8-815b5e399bd2 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:06.353488 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c187bf51-c5c7-4677-ad59-8c5dc9fc761b None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:06.353488 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c187bf51-c5c7-4677-ad59-8c5dc9fc761b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:06.353633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-fab28717-921f-4bb0-91e8-815b5e399bd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:06.353706 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c187bf51-c5c7-4677-ad59-8c5dc9fc761b None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:06.368637 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:06.374498 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 512/1040] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 4 00:48:06 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:06.375720 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c187bf51-c5c7-4677-ad59-8c5dc9fc761b None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:06.442590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 529/1041] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:06 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:48:06.490341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c187bf51-c5c7-4677-ad59-8c5dc9fc761b None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:06.526434 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e0f1c5-8f12-4020-91a9-b3bc7d3bdc2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:06.527063 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e0f1c5-8f12-4020-91a9-b3bc7d3bdc2c None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:06.527390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e0f1c5-8f12-4020-91a9-b3bc7d3bdc2c None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:06.527635 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e0f1c5-8f12-4020-91a9-b3bc7d3bdc2c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:06.587678 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:06.603522 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e0f1c5-8f12-4020-91a9-b3bc7d3bdc2c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:06.604954 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e0f1c5-8f12-4020-91a9-b3bc7d3bdc2c None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:06.704356 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 513/1042] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:48:06 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 04 00:48:06.709906 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab28717-921f-4bb0-91e8-815b5e399bd2 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:06.720043 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e0f1c5-8f12-4020-91a9-b3bc7d3bdc2c None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:06.805639 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:06.810422 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab28717-921f-4bb0-91e8-815b5e399bd2 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:06.822944 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e0f1c5-8f12-4020-91a9-b3bc7d3bdc2c None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:06.845777 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b23dbc-aada-49df-8976-88b51de9eb75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:06.846403 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b23dbc-aada-49df-8976-88b51de9eb75 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:06.846679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b23dbc-aada-49df-8976-88b51de9eb75 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:06.846946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b23dbc-aada-49df-8976-88b51de9eb75 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:06.856944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b23dbc-aada-49df-8976-88b51de9eb75 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:06.858406 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b23dbc-aada-49df-8976-88b51de9eb75 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:06.860886 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea5c114-5ab8-4c5f-a44c-ee7dfb9f5397 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:06.861607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea5c114-5ab8-4c5f-a44c-ee7dfb9f5397 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:06.861946 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea5c114-5ab8-4c5f-a44c-ee7dfb9f5397 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:06.862278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea5c114-5ab8-4c5f-a44c-ee7dfb9f5397 None admin] PATH_INFO: `/v3/services/image` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:06.862923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea5c114-5ab8-4c5f-a44c-ee7dfb9f5397 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:06.864320 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea5c114-5ab8-4c5f-a44c-ee7dfb9f5397 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:06.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-bea5c114-5ab8-4c5f-a44c-ee7dfb9f5397 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 04 00:48:06.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:06.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:48:06.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:48:06.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:48:06.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:48:06.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:48:06.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:48:06.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 04 00:48:06.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:06.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:48:06.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:06.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:06.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:06.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:06.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:06.873062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:06.874507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:06.874507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:06.874507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:06.874507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:06.874507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:06.874507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:06.874507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:48:06.874507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:48:06.874507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:48:06.874507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:48:06.874507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:06.874507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:06.874507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:06.874507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:06.874507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:06.874507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:06.875384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:06.875384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:06.875384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:06.875384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:06.875384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:06.875384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:06.875384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:06.875384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:06.875384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:06.875384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:06.875384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:48:06.875384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:48:06.875384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 04 00:48:06.875384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-06b23dbc-aada-49df-8976-88b51de9eb75 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:48:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:48:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:48:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:48:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:48:06.877038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:06.877038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:06.877038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:06.877038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:06.877038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:06.877038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:06.877038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:06.877038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:06.877038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:06.877038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:06.877038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:06.877038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:06.877038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:06.877038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:06.877038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:06.877038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:06.877038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:48:06.878740 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:48:06.878740 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:48:06.878740 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:48:06.878740 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 530/1043] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:48:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:48:06.881120 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 514/1044] 10.222.0.42 () {56 vars in 1105 bytes} [Sun Sep 4 00:48:06 2022] GET /identity/v3/services/image => generated 86 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:48:06.881249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b23dbc-aada-49df-8976-88b51de9eb75 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:06.882998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:06.885396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea5c114-5ab8-4c5f-a44c-ee7dfb9f5397 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:07.000171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b23dbc-aada-49df-8976-88b51de9eb75 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:07.006738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea5c114-5ab8-4c5f-a44c-ee7dfb9f5397 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:07.038414 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94525107-7cf2-4ce9-b2b0-63c03c7a017c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:07.039012 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94525107-7cf2-4ce9-b2b0-63c03c7a017c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:07.039291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94525107-7cf2-4ce9-b2b0-63c03c7a017c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:07.039563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94525107-7cf2-4ce9-b2b0-63c03c7a017c None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:07.040200 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94525107-7cf2-4ce9-b2b0-63c03c7a017c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:07.040498 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94525107-7cf2-4ce9-b2b0-63c03c7a017c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:07.041894 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94525107-7cf2-4ce9-b2b0-63c03c7a017c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:07.042553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c918f1d4-996f-43cc-943c-1c4b71d5813e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:07.043342 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c918f1d4-996f-43cc-943c-1c4b71d5813e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:07.043572 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c918f1d4-996f-43cc-943c-1c4b71d5813e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:07.043913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c918f1d4-996f-43cc-943c-1c4b71d5813e None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:07.044557 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c918f1d4-996f-43cc-943c-1c4b71d5813e None admin] RBAC: Adding query filter params (name=image) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:07.044867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c918f1d4-996f-43cc-943c-1c4b71d5813e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:07.046255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c918f1d4-996f-43cc-943c-1c4b71d5813e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:07.051340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 515/1045] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:48:06 2022] GET /identity/v3/services?name=image => generated 122 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:07.058944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 531/1046] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:48:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:07.059426 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c918f1d4-996f-43cc-943c-1c4b71d5813e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:07.060149 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94525107-7cf2-4ce9-b2b0-63c03c7a017c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:07.073192 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:07.124574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:07.165499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94525107-7cf2-4ce9-b2b0-63c03c7a017c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:07.166748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c918f1d4-996f-43cc-943c-1c4b71d5813e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:07.202093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d98534f-1aac-45cb-a49c-9dd751a413f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:07.202616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83c6b96-0fb2-45b8-a790-4a48f2466010 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:07.202754 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d98534f-1aac-45cb-a49c-9dd751a413f7 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:07.202987 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d98534f-1aac-45cb-a49c-9dd751a413f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:07.203261 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d98534f-1aac-45cb-a49c-9dd751a413f7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:07.203355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83c6b96-0fb2-45b8-a790-4a48f2466010 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:07.203544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83c6b96-0fb2-45b8-a790-4a48f2466010 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:07.204378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83c6b96-0fb2-45b8-a790-4a48f2466010 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:07.205005 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83c6b96-0fb2-45b8-a790-4a48f2466010 None admin] RBAC: Adding query filter params (type=image) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:07.205391 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83c6b96-0fb2-45b8-a790-4a48f2466010 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:07.206753 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83c6b96-0fb2-45b8-a790-4a48f2466010 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:07.210910 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d98534f-1aac-45cb-a49c-9dd751a413f7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:07.211855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 516/1047] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:48:07 2022] GET /identity/v3/services?type=image => generated 350 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:07.216806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d98534f-1aac-45cb-a49c-9dd751a413f7 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:07.217516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83c6b96-0fb2-45b8-a790-4a48f2466010 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:07.225671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-9d98534f-1aac-45cb-a49c-9dd751a413f7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:48:07.225671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:07.225671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:07.225671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:07.225671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:07.225671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:07.225671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:07.225671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:07.225671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:07.225671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:07.225671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:07.225671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:07.225671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:48:07.225671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:48:07.225671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:48:07.226537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:48:07.226537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:07.226537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:07.226537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:07.226537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:07.226537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:07.226537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:07.226537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:07.226537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:07.226537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:07.226537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:07.226537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:07.226537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:07.226537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:07.226537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:07.226537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:07.226537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:07.227419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:48:07.227419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:48:07.227419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:48:07.227419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:48:07.227419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:48:07.227419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:48:07.227419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:48:07.227419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:48:07.227419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 532/1048] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:48:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:48:07.232371 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d98534f-1aac-45cb-a49c-9dd751a413f7 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:07.304225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:07.328323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83c6b96-0fb2-45b8-a790-4a48f2466010 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:07.354853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d98534f-1aac-45cb-a49c-9dd751a413f7 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:07.366545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bcaaa0-1044-47f2-9fe8-081c179fb728 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:07.367157 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bcaaa0-1044-47f2-9fe8-081c179fb728 None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:07.367801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bcaaa0-1044-47f2-9fe8-081c179fb728 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:07.367801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bcaaa0-1044-47f2-9fe8-081c179fb728 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:07.368371 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bcaaa0-1044-47f2-9fe8-081c179fb728 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:07.369730 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bcaaa0-1044-47f2-9fe8-081c179fb728 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:07.382973 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:07.394761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91864763-dc1e-4557-b47f-0708c8a0ea38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:07.395386 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91864763-dc1e-4557-b47f-0708c8a0ea38 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:07.395660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91864763-dc1e-4557-b47f-0708c8a0ea38 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:07.395925 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91864763-dc1e-4557-b47f-0708c8a0ea38 None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:07.396698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91864763-dc1e-4557-b47f-0708c8a0ea38 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:07.396999 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91864763-dc1e-4557-b47f-0708c8a0ea38 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:07.398443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91864763-dc1e-4557-b47f-0708c8a0ea38 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:07.410912 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 533/1049] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:48:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:07.417015 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91864763-dc1e-4557-b47f-0708c8a0ea38 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:07.467047 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 517/1050] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 00:48:07 2022] POST /identity/v3/endpoints => generated 327 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 00:48:07.472956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a4d97a-7343-4207-91ee-87a857fe2cc1 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:07.473363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a4d97a-7343-4207-91ee-87a857fe2cc1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:07.473650 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a4d97a-7343-4207-91ee-87a857fe2cc1 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:07.475038 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 518/1051] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:07 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:07.480899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f1e610-ceec-426a-aad8-081e40c98788 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:07.481220 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f1e610-ceec-426a-aad8-081e40c98788 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:07.481502 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f1e610-ceec-426a-aad8-081e40c98788 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:07.501234 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-87f1e610-ceec-426a-aad8-081e40c98788 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:07.502576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-87f1e610-ceec-426a-aad8-081e40c98788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:07.531277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91864763-dc1e-4557-b47f-0708c8a0ea38 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:07.582988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 519/1052] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:48:07.583180 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17933746-4d1e-46b1-b3f7-e8aab21dc1b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:07.583682 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17933746-4d1e-46b1-b3f7-e8aab21dc1b5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:07.584100 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17933746-4d1e-46b1-b3f7-e8aab21dc1b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:07.584436 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17933746-4d1e-46b1-b3f7-e8aab21dc1b5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:07.585287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17933746-4d1e-46b1-b3f7-e8aab21dc1b5 None admin] RBAC: Adding query filter params (role.id=ace99e88214440f19edb734e93392300, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=57c88f16c7df40adb917f49804fda21f) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:07.585658 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17933746-4d1e-46b1-b3f7-e8aab21dc1b5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:07.587171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17933746-4d1e-46b1-b3f7-e8aab21dc1b5 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:07.600482 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 534/1053] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:48:07 2022] GET /identity/v3/role_assignments?user.id=57c88f16c7df40adb917f49804fda21f&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=ace99e88214440f19edb734e93392300 => generated 259 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:07.857361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f1e610-ceec-426a-aad8-081e40c98788 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:07.915426 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:07.957526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f1e610-ceec-426a-aad8-081e40c98788 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:07.993427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384a0c82-a3dd-4326-a335-85d08b106ba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:07.994062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384a0c82-a3dd-4326-a335-85d08b106ba6 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:07.994344 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384a0c82-a3dd-4326-a335-85d08b106ba6 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:07.994618 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384a0c82-a3dd-4326-a335-85d08b106ba6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:08.005363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384a0c82-a3dd-4326-a335-85d08b106ba6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:48:08.010138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-384a0c82-a3dd-4326-a335-85d08b106ba6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:48:08.010138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:08.010138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:08.010138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:08.010138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:08.010138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:08.010138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:08.010138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:08.010138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:08.010138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:08.010138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:08.010138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:08.010138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:48:08.010138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:48:08.010138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:48:08.010138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:48:08.011054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:48:08.011054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:48:08.011054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:48:08.011054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:48:08.011054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:08.011054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:08.011054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:08.011054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:08.011054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:08.011054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:08.011054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:08.011054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:08.011054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:08.011054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:08.011054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:08.011054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:08.011919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:08.011919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:08.011919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:08.011919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:08.011919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:48:08.011919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:48:08.011919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:48:08.011919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:48:08.011919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:48:08.011919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:48:08.011919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:48:08.011919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:08.016294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 520/1054] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:48:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:48:08.017444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17933746-4d1e-46b1-b3f7-e8aab21dc1b5 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:08.064060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:08.122860 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17933746-4d1e-46b1-b3f7-e8aab21dc1b5 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:08.157258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd52e44-8ebe-4300-9f7e-d6ccc8e5a786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:08.158063 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd52e44-8ebe-4300-9f7e-d6ccc8e5a786 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:08.158346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd52e44-8ebe-4300-9f7e-d6ccc8e5a786 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:08.158621 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd52e44-8ebe-4300-9f7e-d6ccc8e5a786 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:08.159286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd52e44-8ebe-4300-9f7e-d6ccc8e5a786 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:08.159591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd52e44-8ebe-4300-9f7e-d6ccc8e5a786 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:08.160956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd52e44-8ebe-4300-9f7e-d6ccc8e5a786 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:08.169655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 535/1055] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:48:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:08.176064 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384a0c82-a3dd-4326-a335-85d08b106ba6 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:08.276995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384a0c82-a3dd-4326-a335-85d08b106ba6 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:08.294663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:08.312158 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e5c17-0ced-4ef9-9438-8fc2a81edc9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:08.312783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e5c17-0ced-4ef9-9438-8fc2a81edc9e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:08.313788 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e5c17-0ced-4ef9-9438-8fc2a81edc9e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:08.314171 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e5c17-0ced-4ef9-9438-8fc2a81edc9e None admin] PATH_INFO: `/v3/users/8781b422f340440a87cc89b8fb2d20dc` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:08.327428 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9e5c17-0ced-4ef9-9438-8fc2a81edc9e None admin] RBAC: Authorizing `identity:get_user(user_id=8781b422f340440a87cc89b8fb2d20dc)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:08.328967 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9e5c17-0ced-4ef9-9438-8fc2a81edc9e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:08.330882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 521/1056] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:48:08 2022] GET /identity/v3/users/8781b422f340440a87cc89b8fb2d20dc => generated 285 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:08.336571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd52e44-8ebe-4300-9f7e-d6ccc8e5a786 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:08.442017 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd52e44-8ebe-4300-9f7e-d6ccc8e5a786 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:08.444333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:08.480420 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a428d48f-df70-4146-8ce5-854bb6841e6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:08.481024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a428d48f-df70-4146-8ce5-854bb6841e6b None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:08.481341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a428d48f-df70-4146-8ce5-854bb6841e6b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:08.481614 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a428d48f-df70-4146-8ce5-854bb6841e6b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:08.551636 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a428d48f-df70-4146-8ce5-854bb6841e6b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:08.553088 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a428d48f-df70-4146-8ce5-854bb6841e6b None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:08.663953 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 536/1057] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:48:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 04 00:48:08.665687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e5c17-0ced-4ef9-9438-8fc2a81edc9e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:08.767951 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e5c17-0ced-4ef9-9438-8fc2a81edc9e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:08.800558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb3ad20-a03f-419e-81b3-b47738a380f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:08.801202 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb3ad20-a03f-419e-81b3-b47738a380f6 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:08.801486 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb3ad20-a03f-419e-81b3-b47738a380f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:08.801753 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb3ad20-a03f-419e-81b3-b47738a380f6 None admin] PATH_INFO: `/v3/projects/a46e66ec57f04506a481aba0e2838c2c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:08.802846 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb3ad20-a03f-419e-81b3-b47738a380f6 None admin] RBAC: Authorizing `identity:get_project(project_id=a46e66ec57f04506a481aba0e2838c2c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:08.804250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb3ad20-a03f-419e-81b3-b47738a380f6 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:08.806037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 522/1058] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:48:08 2022] GET /identity/v3/projects/a46e66ec57f04506a481aba0e2838c2c => generated 311 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:08.812014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a428d48f-df70-4146-8ce5-854bb6841e6b None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:08.815794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:08.911375 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a428d48f-df70-4146-8ce5-854bb6841e6b None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:08.945393 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652620a2-abc8-457a-914b-3219c6e7080b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:08.946013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652620a2-abc8-457a-914b-3219c6e7080b None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:08.946288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652620a2-abc8-457a-914b-3219c6e7080b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:08.946559 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652620a2-abc8-457a-914b-3219c6e7080b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:08.947275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652620a2-abc8-457a-914b-3219c6e7080b None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=a46e66ec57f04506a481aba0e2838c2c, user.id=8781b422f340440a87cc89b8fb2d20dc) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:08.947579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652620a2-abc8-457a-914b-3219c6e7080b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:08.948984 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652620a2-abc8-457a-914b-3219c6e7080b None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:08.955078 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 537/1059] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:48:08 2022] GET /identity/v3/role_assignments?user.id=8781b422f340440a87cc89b8fb2d20dc&scope.project.id=a46e66ec57f04506a481aba0e2838c2c&role.id=19c006412bec45dc8febd7cc86b9678f => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:09.444070 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fe649f-6691-43b7-ad15-54fefebd732b None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:09.444614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fe649f-6691-43b7-ad15-54fefebd732b None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:09.444614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fe649f-6691-43b7-ad15-54fefebd732b None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:09.446031 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 523/1060] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:09.451123 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487eae72-0123-48f2-a089-628b9b4c618e None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:09.451507 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487eae72-0123-48f2-a089-628b9b4c618e None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:09.451665 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487eae72-0123-48f2-a089-628b9b4c618e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:09.473290 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-487eae72-0123-48f2-a089-628b9b4c618e None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:09.474522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-487eae72-0123-48f2-a089-628b9b4c618e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:09.514294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febff9c5-9c12-490b-af89-fef860d2b386 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:09.514633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febff9c5-9c12-490b-af89-fef860d2b386 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:09.514853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febff9c5-9c12-490b-af89-fef860d2b386 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:09.516902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 524/1061] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:48:09.521606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e4b442-98e9-4303-a191-0515aab32f2c None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:09.521881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e4b442-98e9-4303-a191-0515aab32f2c None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:09.522153 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e4b442-98e9-4303-a191-0515aab32f2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:09.540252 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:09.540431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-65e4b442-98e9-4303-a191-0515aab32f2c None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:09.541592 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-65e4b442-98e9-4303-a191-0515aab32f2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:09.551394 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 538/1062] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:09.585029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:09.610719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 525/1063] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:48:09.823814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487eae72-0123-48f2-a089-628b9b4c618e None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:09.877478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e4b442-98e9-4303-a191-0515aab32f2c None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:09.929483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487eae72-0123-48f2-a089-628b9b4c618e None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:09.972797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ed19c1-be45-4592-b8cd-1d61c1243487 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:09.973433 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ed19c1-be45-4592-b8cd-1d61c1243487 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:09.973713 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ed19c1-be45-4592-b8cd-1d61c1243487 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:09.973980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ed19c1-be45-4592-b8cd-1d61c1243487 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:09.983049 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e4b442-98e9-4303-a191-0515aab32f2c None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:10.028127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d42a50-9ca1-4196-89da-c3bd06fcb62d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:10.028768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d42a50-9ca1-4196-89da-c3bd06fcb62d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:10.029052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d42a50-9ca1-4196-89da-c3bd06fcb62d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:10.029358 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d42a50-9ca1-4196-89da-c3bd06fcb62d None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:10.040769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d42a50-9ca1-4196-89da-c3bd06fcb62d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:48:10.045709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-c1d42a50-9ca1-4196-89da-c3bd06fcb62d None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 04 00:48:10.045709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:10.045709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:10.045709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:10.045709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:10.045709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:10.045709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:10.045709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:10.045709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:10.045709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:10.045709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:10.045709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:10.045709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:48:10.045709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:48:10.045709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:48:10.045709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:48:10.046643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:48:10.046643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:48:10.046643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:48:10.046643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:48:10.046643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:10.046643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:10.046643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:10.046643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:10.046643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:10.046643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:10.046643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:10.046643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:10.046643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:10.046643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:10.046643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:10.046643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:10.047663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:10.047663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:10.047663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:10.047663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:10.047663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:48:10.047663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:48:10.047663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:48:10.047663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:48:10.047663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:48:10.047663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:48:10.047663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 04 00:48:10.047663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:10.047663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 526/1064] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Sep 4 00:48:09 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:48:10.052757 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d42a50-9ca1-4196-89da-c3bd06fcb62d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:10.053248 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:10.056849 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ed19c1-be45-4592-b8cd-1d61c1243487 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:10.058385 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ed19c1-be45-4592-b8cd-1d61c1243487 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:10.091503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:10.161396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d42a50-9ca1-4196-89da-c3bd06fcb62d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:10.162189 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 539/1065] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:48:09 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 04 00:48:10.169115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ed19c1-be45-4592-b8cd-1d61c1243487 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:10.196779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c66d3b-fb2d-4039-9ee8-bce3a4828e92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:10.197487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c66d3b-fb2d-4039-9ee8-bce3a4828e92 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:10.197754 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c66d3b-fb2d-4039-9ee8-bce3a4828e92 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:10.198026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c66d3b-fb2d-4039-9ee8-bce3a4828e92 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:10.198696 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c66d3b-fb2d-4039-9ee8-bce3a4828e92 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:10.199005 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c66d3b-fb2d-4039-9ee8-bce3a4828e92 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:10.200360 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c66d3b-fb2d-4039-9ee8-bce3a4828e92 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:10.213718 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 527/1066] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Sep 4 00:48:10 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:10.214392 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c66d3b-fb2d-4039-9ee8-bce3a4828e92 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:10.268949 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:10.273486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ed19c1-be45-4592-b8cd-1d61c1243487 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:10.314507 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b447cc-1f2c-4b37-96a4-4bf9d31b0676 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:10.315143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b447cc-1f2c-4b37-96a4-4bf9d31b0676 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:10.315422 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b447cc-1f2c-4b37-96a4-4bf9d31b0676 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:10.315697 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b447cc-1f2c-4b37-96a4-4bf9d31b0676 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:10.316109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:10.321823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c66d3b-fb2d-4039-9ee8-bce3a4828e92 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:10.326786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b447cc-1f2c-4b37-96a4-4bf9d31b0676 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:10.329543 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b447cc-1f2c-4b37-96a4-4bf9d31b0676 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:10.345522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-61b447cc-1f2c-4b37-96a4-4bf9d31b0676 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:48:10.345522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:10.345522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:10.345522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:10.345522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:10.345522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:10.345522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:10.345522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:10.345522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:10.345522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:10.345522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:10.345522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:10.345522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:48:10.345522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:48:10.345522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:48:10.345522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:48:10.346461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:10.346461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:10.346461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:10.346461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:10.346461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:10.346461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:10.346461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:10.346461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:10.346461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:10.346461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:10.346461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:10.346461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:10.346461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:10.346461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:10.346461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:10.346461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:10.346461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:48:10.347333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:48:10.347333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:48:10.347333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:48:10.347333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 540/1067] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:48:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:48:10.352429 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b447cc-1f2c-4b37-96a4-4bf9d31b0676 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:10.363272 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac411e3-1cc4-408a-9c5c-cb398c90c455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:10.364135 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac411e3-1cc4-408a-9c5c-cb398c90c455 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:10.364776 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac411e3-1cc4-408a-9c5c-cb398c90c455 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:10.365050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac411e3-1cc4-408a-9c5c-cb398c90c455 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:10.377242 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac411e3-1cc4-408a-9c5c-cb398c90c455 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:10.378677 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac411e3-1cc4-408a-9c5c-cb398c90c455 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:10.394093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-7ac411e3-1cc4-408a-9c5c-cb398c90c455 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:48:10.394093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:10.394093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:10.394093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:10.394093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:10.394093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:10.394093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:10.394093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:10.394093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:10.394093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:10.394093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:10.394093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:10.394093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:48:10.394093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:48:10.394093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:48:10.394093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:48:10.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:10.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:10.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:10.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:10.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:10.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:10.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:10.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:10.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:10.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:10.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:10.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:10.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:10.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:10.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:10.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:10.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:48:10.395943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:48:10.395943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:48:10.395943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:10.400032 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 528/1068] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:48:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:48:10.400894 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac411e3-1cc4-408a-9c5c-cb398c90c455 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:10.468132 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b447cc-1f2c-4b37-96a4-4bf9d31b0676 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:10.504084 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d5317f-e6c4-4d54-9dfa-15e96a922933 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:10.504729 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d5317f-e6c4-4d54-9dfa-15e96a922933 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:10.504979 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d5317f-e6c4-4d54-9dfa-15e96a922933 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:10.505274 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d5317f-e6c4-4d54-9dfa-15e96a922933 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:10.505922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d5317f-e6c4-4d54-9dfa-15e96a922933 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:10.506237 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d5317f-e6c4-4d54-9dfa-15e96a922933 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:10.507620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d5317f-e6c4-4d54-9dfa-15e96a922933 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:10.512957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac411e3-1cc4-408a-9c5c-cb398c90c455 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:10.520633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 541/1069] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:48:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:10.552681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a64204-6ba9-4ad6-b027-e17a065cee00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:10.553350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a64204-6ba9-4ad6-b027-e17a065cee00 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:10.553628 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a64204-6ba9-4ad6-b027-e17a065cee00 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:10.553901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a64204-6ba9-4ad6-b027-e17a065cee00 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:10.554577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a64204-6ba9-4ad6-b027-e17a065cee00 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:10.554886 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a64204-6ba9-4ad6-b027-e17a065cee00 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:10.556281 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a64204-6ba9-4ad6-b027-e17a065cee00 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:10.573547 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 529/1070] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:48:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:10.579972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d5317f-e6c4-4d54-9dfa-15e96a922933 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:10.593430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:10.693178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d5317f-e6c4-4d54-9dfa-15e96a922933 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:10.728966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9120468-5c2a-48bb-b566-a798b00ff1ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:10.729624 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9120468-5c2a-48bb-b566-a798b00ff1ad None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:10.729898 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9120468-5c2a-48bb-b566-a798b00ff1ad None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:10.730168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9120468-5c2a-48bb-b566-a798b00ff1ad None admin] PATH_INFO: `/v3/users/nova` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:10.737670 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9120468-5c2a-48bb-b566-a798b00ff1ad None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:10.739103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9120468-5c2a-48bb-b566-a798b00ff1ad None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:10.750857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-f9120468-5c2a-48bb-b566-a798b00ff1ad None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:48:10.750857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:10.750857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:10.750857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:10.750857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:10.750857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:10.750857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:10.750857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:10.750857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:10.750857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:10.750857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:10.750857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:10.750857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:48:10.750857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:48:10.750857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:48:10.750857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:10.751800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:10.753243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:10.753243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:10.753243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:48:10.753243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:48:10.753243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:48:10.753243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:48:10.753243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:48:10.753243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:48:10.753243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:48:10.753243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:48:10.753243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 542/1071] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 4 00:48:10 2022] GET /identity/v3/users/nova => generated 82 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:48:10.758129 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a64204-6ba9-4ad6-b027-e17a065cee00 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:10.770776 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:10.864330 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992b3221-e6e4-489c-ada3-c11c1d3906db None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:10.864611 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992b3221-e6e4-489c-ada3-c11c1d3906db None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:10.864838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a64204-6ba9-4ad6-b027-e17a065cee00 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:10.864940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992b3221-e6e4-489c-ada3-c11c1d3906db None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:10.866242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 543/1072] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:10.870865 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1788949f-60f4-458a-9495-1101780e2590 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:10.871145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1788949f-60f4-458a-9495-1101780e2590 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:10.871418 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1788949f-60f4-458a-9495-1101780e2590 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:10.890133 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-1788949f-60f4-458a-9495-1101780e2590 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:10.891928 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-1788949f-60f4-458a-9495-1101780e2590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:10.900955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1e24a9-573b-4796-a34b-823d14aca9f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:10.901587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1e24a9-573b-4796-a34b-823d14aca9f0 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:10.901861 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1e24a9-573b-4796-a34b-823d14aca9f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:10.902126 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1e24a9-573b-4796-a34b-823d14aca9f0 None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:10.902857 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1e24a9-573b-4796-a34b-823d14aca9f0 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:10.903159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1e24a9-573b-4796-a34b-823d14aca9f0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:10.904563 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1e24a9-573b-4796-a34b-823d14aca9f0 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:10.921652 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 530/1073] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 4 00:48:10 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:10.926873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1e24a9-573b-4796-a34b-823d14aca9f0 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:10.955546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:10.959997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 544/1074] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:48:10.990319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:11.023189 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1e24a9-573b-4796-a34b-823d14aca9f0 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:11.056112 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07cf47ef-3c52-4792-bb76-ff626806aad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:11.056726 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cf47ef-3c52-4792-bb76-ff626806aad1 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:11.057004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cf47ef-3c52-4792-bb76-ff626806aad1 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:11.057303 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cf47ef-3c52-4792-bb76-ff626806aad1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:11.067474 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07cf47ef-3c52-4792-bb76-ff626806aad1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:11.068885 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07cf47ef-3c52-4792-bb76-ff626806aad1 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:11.082419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-07cf47ef-3c52-4792-bb76-ff626806aad1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:48:11.082419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:11.082419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:11.082419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:11.082419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:11.082419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:11.082419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:11.082419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:11.082419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:11.082419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:11.082419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:11.082419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:11.082419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:48:11.082419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:48:11.082419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:48:11.082419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:48:11.083581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:11.083581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:11.083581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:11.083581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:11.083581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:11.083581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:11.083581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:11.083581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:11.083581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:11.083581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:11.083581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:11.083581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:11.083581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:11.083581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:11.083581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:11.083581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:11.083581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:48:11.084962 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:48:11.084962 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:48:11.084962 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:11.084962 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 531/1075] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:48:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:48:11.089257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1788949f-60f4-458a-9495-1101780e2590 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:11.191380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1788949f-60f4-458a-9495-1101780e2590 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:11.226531 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a68d98-974c-40dd-9ea9-470beb2a0d79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:11.226768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07cf47ef-3c52-4792-bb76-ff626806aad1 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:11.227073 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a68d98-974c-40dd-9ea9-470beb2a0d79 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:11.227347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a68d98-974c-40dd-9ea9-470beb2a0d79 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:11.227618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a68d98-974c-40dd-9ea9-470beb2a0d79 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:11.228265 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a68d98-974c-40dd-9ea9-470beb2a0d79 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:11.228567 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a68d98-974c-40dd-9ea9-470beb2a0d79 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:11.230055 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a68d98-974c-40dd-9ea9-470beb2a0d79 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:11.242135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 545/1076] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:48:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:11.247699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a68d98-974c-40dd-9ea9-470beb2a0d79 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:11.305403 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:11.330354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07cf47ef-3c52-4792-bb76-ff626806aad1 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:11.346634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a68d98-974c-40dd-9ea9-470beb2a0d79 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:11.347159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:11.363959 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96acf2bf-8798-4866-89b7-9f3938abe683 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:11.364585 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96acf2bf-8798-4866-89b7-9f3938abe683 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:11.364866 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96acf2bf-8798-4866-89b7-9f3938abe683 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:11.365168 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96acf2bf-8798-4866-89b7-9f3938abe683 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:11.376924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96acf2bf-8798-4866-89b7-9f3938abe683 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:48:11.380350 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401109e4-59a3-4554-9d8c-f7c33a8d401f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:11.380972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401109e4-59a3-4554-9d8c-f7c33a8d401f None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:11.381288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401109e4-59a3-4554-9d8c-f7c33a8d401f None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:11.381561 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401109e4-59a3-4554-9d8c-f7c33a8d401f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:11.381703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-96acf2bf-8798-4866-89b7-9f3938abe683 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:48:11.381703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:11.381703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:11.381703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:11.381703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:11.381703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:11.381703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:11.381703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:11.381703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:11.381703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:11.381703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:11.381703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:11.381703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:48:11.381703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:48:11.381703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:48:11.381703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:48:11.382582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:48:11.382582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:48:11.382582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:48:11.382582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:48:11.382582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:11.382582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:11.382582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:11.382582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:11.382582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:11.382582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:11.382582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:11.382582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:11.382582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:11.382582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:11.382582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:11.382582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:11.383521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:11.383521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:11.383521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:11.383521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:11.383521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:48:11.383521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:48:11.383521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:48:11.383521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:48:11.383521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:48:11.383521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:48:11.383521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:48:11.383521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:11.383521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 532/1077] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:48:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:48:11.388472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401109e4-59a3-4554-9d8c-f7c33a8d401f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:11.388619 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96acf2bf-8798-4866-89b7-9f3938abe683 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:11.394926 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401109e4-59a3-4554-9d8c-f7c33a8d401f None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:11.401306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-401109e4-59a3-4554-9d8c-f7c33a8d401f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:48:11.401306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:11.401306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:11.401306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:11.401306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:11.401306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:11.401306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:11.401306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:11.401306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:11.401306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:11.401306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:11.401306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:11.401306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:48:11.401306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:48:11.401306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:48:11.402353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:48:11.402353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:11.402353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:11.402353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:11.402353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:11.402353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:11.402353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:11.402353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:11.402353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:11.402353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:11.402353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:11.402353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:11.402353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:11.402353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:11.402353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:11.402353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:11.402353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:11.403428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:48:11.403428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:48:11.403428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:48:11.403428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:48:11.403428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:48:11.403428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:48:11.403428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:48:11.403428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:48:11.403428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 546/1078] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:48:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:48:11.408401 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401109e4-59a3-4554-9d8c-f7c33a8d401f None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:11.494540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96acf2bf-8798-4866-89b7-9f3938abe683 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:11.509031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401109e4-59a3-4554-9d8c-f7c33a8d401f None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:11.527943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:11.528476 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6338919-c684-4c46-8e1b-a715df296318 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:11.529106 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6338919-c684-4c46-8e1b-a715df296318 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:11.529391 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6338919-c684-4c46-8e1b-a715df296318 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:11.529669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6338919-c684-4c46-8e1b-a715df296318 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:11.530344 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6338919-c684-4c46-8e1b-a715df296318 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:11.530651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6338919-c684-4c46-8e1b-a715df296318 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:11.532374 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6338919-c684-4c46-8e1b-a715df296318 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:11.540850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 533/1079] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:48:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:11.542032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8f4947-ffeb-443d-9890-208682ea3ff3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:11.542645 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8f4947-ffeb-443d-9890-208682ea3ff3 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:11.542924 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8f4947-ffeb-443d-9890-208682ea3ff3 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:11.543191 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8f4947-ffeb-443d-9890-208682ea3ff3 None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:11.543879 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8f4947-ffeb-443d-9890-208682ea3ff3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:11.544185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8f4947-ffeb-443d-9890-208682ea3ff3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:11.546101 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8f4947-ffeb-443d-9890-208682ea3ff3 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:11.546596 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6338919-c684-4c46-8e1b-a715df296318 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:11.559362 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 547/1080] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:48:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:11.565304 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8f4947-ffeb-443d-9890-208682ea3ff3 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:11.589540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:11.647264 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6338919-c684-4c46-8e1b-a715df296318 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:11.666389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8f4947-ffeb-443d-9890-208682ea3ff3 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:11.681871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b575e1c3-4aa4-495d-bf90-ea5aeb8c4044 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:11.682502 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b575e1c3-4aa4-495d-bf90-ea5aeb8c4044 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:11.682778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b575e1c3-4aa4-495d-bf90-ea5aeb8c4044 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:11.683055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b575e1c3-4aa4-495d-bf90-ea5aeb8c4044 None admin] PATH_INFO: `/v3/users/8781b422f340440a87cc89b8fb2d20dc` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:11.684173 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b575e1c3-4aa4-495d-bf90-ea5aeb8c4044 None admin] RBAC: Authorizing `identity:get_user(user_id=8781b422f340440a87cc89b8fb2d20dc)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:11.685620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b575e1c3-4aa4-495d-bf90-ea5aeb8c4044 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:11.687174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 534/1081] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:48:11 2022] GET /identity/v3/users/8781b422f340440a87cc89b8fb2d20dc => generated 285 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:11.692931 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b575e1c3-4aa4-495d-bf90-ea5aeb8c4044 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:11.702399 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8624eb93-2e0f-4f03-8e63-2a5cb089a926 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:11.703070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8624eb93-2e0f-4f03-8e63-2a5cb089a926 None admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:11.703527 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8624eb93-2e0f-4f03-8e63-2a5cb089a926 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:11.703934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8624eb93-2e0f-4f03-8e63-2a5cb089a926 None admin] PATH_INFO: `/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/57c88f16c7df40adb917f49804fda21f/roles/ace99e88214440f19edb734e93392300` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:11.706363 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8624eb93-2e0f-4f03-8e63-2a5cb089a926 None admin] RBAC: Authorizing `identity:create_grant(project_id=b1f56314d44d4faaa88e735ea196783f, user_id=57c88f16c7df40adb917f49804fda21f, role_id=ace99e88214440f19edb734e93392300)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:11.707799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8624eb93-2e0f-4f03-8e63-2a5cb089a926 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:11.785924 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 548/1082] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:48:11 2022] PUT /identity/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/57c88f16c7df40adb917f49804fda21f/roles/ace99e88214440f19edb734e93392300 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 00:48:11.789183 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b575e1c3-4aa4-495d-bf90-ea5aeb8c4044 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:11.799848 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:11.825811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426164f0-a8dc-4b39-b520-e7138bb93f7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:11.826454 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426164f0-a8dc-4b39-b520-e7138bb93f7f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:11.826730 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426164f0-a8dc-4b39-b520-e7138bb93f7f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:11.827001 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426164f0-a8dc-4b39-b520-e7138bb93f7f None admin] PATH_INFO: `/v3/projects/a46e66ec57f04506a481aba0e2838c2c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:11.828169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426164f0-a8dc-4b39-b520-e7138bb93f7f None admin] RBAC: Authorizing `identity:get_project(project_id=a46e66ec57f04506a481aba0e2838c2c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:11.829828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426164f0-a8dc-4b39-b520-e7138bb93f7f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:11.831569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 535/1083] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:48:11 2022] GET /identity/v3/projects/a46e66ec57f04506a481aba0e2838c2c => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:11.837954 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8624eb93-2e0f-4f03-8e63-2a5cb089a926 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:11.884789 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:11.945890 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8624eb93-2e0f-4f03-8e63-2a5cb089a926 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:11.980436 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af859fef-7b87-4a6d-8392-891b7e1db8ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:11.981123 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af859fef-7b87-4a6d-8392-891b7e1db8ac None admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:11.981408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af859fef-7b87-4a6d-8392-891b7e1db8ac None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:11.981687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af859fef-7b87-4a6d-8392-891b7e1db8ac None admin] PATH_INFO: `/v3/projects/a46e66ec57f04506a481aba0e2838c2c/users/8781b422f340440a87cc89b8fb2d20dc/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:11.983596 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af859fef-7b87-4a6d-8392-891b7e1db8ac None admin] RBAC: Authorizing `identity:create_grant(project_id=a46e66ec57f04506a481aba0e2838c2c, user_id=8781b422f340440a87cc89b8fb2d20dc, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:11.985618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af859fef-7b87-4a6d-8392-891b7e1db8ac None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:12.060365 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 549/1084] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:48:11 2022] PUT /identity/v3/projects/a46e66ec57f04506a481aba0e2838c2c/users/8781b422f340440a87cc89b8fb2d20dc/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:48:13.675557 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a296503-ccc5-4831-93b7-45d3cd648048 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:13.676107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a296503-ccc5-4831-93b7-45d3cd648048 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:13.676107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a296503-ccc5-4831-93b7-45d3cd648048 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:13.677620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 536/1085] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:13.682509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacd6f43-e4c4-48c9-ac77-6223b0d92ae3 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:13.682928 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacd6f43-e4c4-48c9-ac77-6223b0d92ae3 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:13.683062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacd6f43-e4c4-48c9-ac77-6223b0d92ae3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:13.702110 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-dacd6f43-e4c4-48c9-ac77-6223b0d92ae3 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:13.703322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-dacd6f43-e4c4-48c9-ac77-6223b0d92ae3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:13.770199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 550/1086] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:13.976695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b59c3d-2947-4e42-9d1e-de9b5453e574 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:13.977046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b59c3d-2947-4e42-9d1e-de9b5453e574 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:13.977363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b59c3d-2947-4e42-9d1e-de9b5453e574 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:13.978684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 537/1087] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:13.983916 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1433aa1b-ea23-4174-aabf-c48b0c010fa5 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:13.984207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1433aa1b-ea23-4174-aabf-c48b0c010fa5 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:13.984480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1433aa1b-ea23-4174-aabf-c48b0c010fa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:14.003430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-1433aa1b-ea23-4174-aabf-c48b0c010fa5 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:14.004598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-1433aa1b-ea23-4174-aabf-c48b0c010fa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:14.027852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:14.040505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b59c3d-2947-4e42-9d1e-de9b5453e574 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:14.073241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 551/1088] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:14.142831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b59c3d-2947-4e42-9d1e-de9b5453e574 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:14.175411 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76355fdc-bc01-43b4-a7fe-153bfdf73302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:14.176034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76355fdc-bc01-43b4-a7fe-153bfdf73302 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:14.176317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76355fdc-bc01-43b4-a7fe-153bfdf73302 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:14.176591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76355fdc-bc01-43b4-a7fe-153bfdf73302 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:14.186642 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76355fdc-bc01-43b4-a7fe-153bfdf73302 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:48:14.192051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-76355fdc-bc01-43b4-a7fe-153bfdf73302 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 04 00:48:14.192051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:14.192051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:14.192051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:14.192051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:14.192051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:14.192051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:14.192051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:14.192051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:14.192051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:14.192051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:14.192051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:14.192051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:48:14.192051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:48:14.192051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:48:14.192051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:48:14.192964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:48:14.192964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:48:14.192964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:48:14.192964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:48:14.192964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:14.192964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:14.192964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:14.192964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:14.192964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:14.192964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:14.192964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:14.192964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:14.192964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:14.192964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:14.192964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:14.192964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:14.194030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:14.194030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:14.194030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:14.194030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:14.194030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:48:14.194030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:48:14.194030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:48:14.194030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:48:14.194030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:48:14.194030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:48:14.194030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 04 00:48:14.194030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:14.194030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 538/1089] 10.222.0.42 () {56 vars in 1120 bytes} [Sun Sep 4 00:48:14 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:48:14.199117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1433aa1b-ea23-4174-aabf-c48b0c010fa5 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:14.295049 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1433aa1b-ea23-4174-aabf-c48b0c010fa5 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:14.328509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4407fd74-9b10-44aa-acad-2a18cc7461a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:14.329130 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4407fd74-9b10-44aa-acad-2a18cc7461a8 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:14.329409 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4407fd74-9b10-44aa-acad-2a18cc7461a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:14.329679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4407fd74-9b10-44aa-acad-2a18cc7461a8 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:14.330347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4407fd74-9b10-44aa-acad-2a18cc7461a8 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:14.330649 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4407fd74-9b10-44aa-acad-2a18cc7461a8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:14.332004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4407fd74-9b10-44aa-acad-2a18cc7461a8 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:14.339709 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 552/1090] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Sep 4 00:48:14 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:14.345060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76355fdc-bc01-43b4-a7fe-153bfdf73302 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:14.345211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4407fd74-9b10-44aa-acad-2a18cc7461a8 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:14.370258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:14.381165 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:14.446320 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76355fdc-bc01-43b4-a7fe-153bfdf73302 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:14.446687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4407fd74-9b10-44aa-acad-2a18cc7461a8 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:14.480649 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ec4884-3028-4346-979a-c9f18001d4d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:14.481015 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f0dd9f-7c20-4acf-8ed7-0f96e7a67326 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:14.481306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ec4884-3028-4346-979a-c9f18001d4d2 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:14.481589 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ec4884-3028-4346-979a-c9f18001d4d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:14.481683 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f0dd9f-7c20-4acf-8ed7-0f96e7a67326 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:14.481866 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ec4884-3028-4346-979a-c9f18001d4d2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:14.481970 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f0dd9f-7c20-4acf-8ed7-0f96e7a67326 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:14.482213 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f0dd9f-7c20-4acf-8ed7-0f96e7a67326 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:14.492885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f0dd9f-7c20-4acf-8ed7-0f96e7a67326 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:14.493132 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ec4884-3028-4346-979a-c9f18001d4d2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:48:14.494383 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f0dd9f-7c20-4acf-8ed7-0f96e7a67326 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:14.497961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-a2ec4884-3028-4346-979a-c9f18001d4d2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:48:14.497961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:14.497961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:14.497961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:14.497961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:14.497961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:14.497961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:14.497961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:14.497961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:14.497961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:14.497961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:14.497961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:14.497961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:48:14.497961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:48:14.497961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:48:14.497961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:48:14.499372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:48:14.499372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:48:14.499372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:48:14.499372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:48:14.499372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:14.499372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:14.499372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:14.499372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:14.499372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:14.499372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:14.499372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:14.499372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:14.499372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:14.499372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:14.499372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:14.499372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:14.500570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:14.500570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:14.500570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:14.500570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:14.500570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:48:14.500570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:48:14.500570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:48:14.500570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:48:14.500570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:48:14.500570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:48:14.500570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:48:14.500570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:14.501803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 539/1091] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:48:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:48:14.505501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ec4884-3028-4346-979a-c9f18001d4d2 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:14.509212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-60f0dd9f-7c20-4acf-8ed7-0f96e7a67326 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:48:14.509212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:14.509212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:14.509212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:14.509212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:14.509212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:14.509212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:14.509212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:14.509212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:14.509212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:14.509212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:14.509212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:14.509212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:48:14.509212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:48:14.509212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:48:14.509212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:48:14.510108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:14.510108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:14.510108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:14.510108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:14.510108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:14.510108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:14.510108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:14.510108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:14.510108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:14.510108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:14.510108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:14.510108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:14.510108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:14.510108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:14.510108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:14.510108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:14.510108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:48:14.510993 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:48:14.510993 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:48:14.510993 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:48:14.510993 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 553/1092] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:48:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:48:14.516496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f0dd9f-7c20-4acf-8ed7-0f96e7a67326 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:14.592188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:14.601800 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:14.608001 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ec4884-3028-4346-979a-c9f18001d4d2 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:14.622830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f0dd9f-7c20-4acf-8ed7-0f96e7a67326 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:14.643516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2476249-988e-456e-8f44-183c0228c574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:14.644341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2476249-988e-456e-8f44-183c0228c574 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:14.644612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2476249-988e-456e-8f44-183c0228c574 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:14.644886 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2476249-988e-456e-8f44-183c0228c574 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:14.645578 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2476249-988e-456e-8f44-183c0228c574 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:14.645875 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2476249-988e-456e-8f44-183c0228c574 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:14.647218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2476249-988e-456e-8f44-183c0228c574 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:14.657340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 540/1093] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:48:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:14.658207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13dbcefa-4ea1-4ba8-a931-8aa3d024bc8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:14.658815 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dbcefa-4ea1-4ba8-a931-8aa3d024bc8a None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:14.659091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dbcefa-4ea1-4ba8-a931-8aa3d024bc8a None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:14.659363 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13dbcefa-4ea1-4ba8-a931-8aa3d024bc8a None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:14.660006 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13dbcefa-4ea1-4ba8-a931-8aa3d024bc8a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:14.660306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13dbcefa-4ea1-4ba8-a931-8aa3d024bc8a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:14.661945 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13dbcefa-4ea1-4ba8-a931-8aa3d024bc8a None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:14.662181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2476249-988e-456e-8f44-183c0228c574 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:14.680628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 554/1094] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:48:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:14.681750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13dbcefa-4ea1-4ba8-a931-8aa3d024bc8a None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:14.761517 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2476249-988e-456e-8f44-183c0228c574 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:14.779806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13dbcefa-4ea1-4ba8-a931-8aa3d024bc8a None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:14.794709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a296d84c-7c20-45bf-a08b-e4255a4b7a97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:14.795331 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a296d84c-7c20-45bf-a08b-e4255a4b7a97 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:14.795610 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a296d84c-7c20-45bf-a08b-e4255a4b7a97 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:14.795883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a296d84c-7c20-45bf-a08b-e4255a4b7a97 None admin] PATH_INFO: `/v3/users/8781b422f340440a87cc89b8fb2d20dc` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:14.797001 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a296d84c-7c20-45bf-a08b-e4255a4b7a97 None admin] RBAC: Authorizing `identity:get_user(user_id=8781b422f340440a87cc89b8fb2d20dc)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:14.799151 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a296d84c-7c20-45bf-a08b-e4255a4b7a97 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:14.800715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 541/1095] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:48:14 2022] GET /identity/v3/users/8781b422f340440a87cc89b8fb2d20dc => generated 285 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:14.806619 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a296d84c-7c20-45bf-a08b-e4255a4b7a97 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:14.814654 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a7c85d-02e0-4ae5-878a-1efaafd69095 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:14.815282 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a7c85d-02e0-4ae5-878a-1efaafd69095 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:14.815557 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a7c85d-02e0-4ae5-878a-1efaafd69095 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:14.816235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a7c85d-02e0-4ae5-878a-1efaafd69095 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:14.818595 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:14.824242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a7c85d-02e0-4ae5-878a-1efaafd69095 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:14.825722 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a7c85d-02e0-4ae5-878a-1efaafd69095 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:14.834510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:14.837976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-46a7c85d-02e0-4ae5-878a-1efaafd69095 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:48:14.837976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:14.837976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:14.837976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:14.837976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:14.837976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:14.837976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:14.837976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:14.837976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:14.837976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:14.837976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:14.837976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:14.837976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:48:14.837976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:48:14.837976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:48:14.837976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:14.839117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:14.840125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:14.840125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:14.840125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:48:14.840125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:48:14.840125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:48:14.840125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:48:14.840125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:48:14.840125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:48:14.840125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 04 00:48:14.840125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:48:14.840125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 555/1096] 10.222.0.42 () {56 vars in 1093 bytes} [Sun Sep 4 00:48:14 2022] GET /identity/v3/users/nova => generated 82 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:48:14.845339 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a7c85d-02e0-4ae5-878a-1efaafd69095 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:14.909255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a296d84c-7c20-45bf-a08b-e4255a4b7a97 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:14.942934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5248350-4fd8-456f-b5f5-8a2f647fa40e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:14.943531 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5248350-4fd8-456f-b5f5-8a2f647fa40e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:14.943808 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5248350-4fd8-456f-b5f5-8a2f647fa40e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:14.944078 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5248350-4fd8-456f-b5f5-8a2f647fa40e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:14.944488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a7c85d-02e0-4ae5-878a-1efaafd69095 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:14.978068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c65456d-1736-4e44-b02f-bdaad613572e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:14.978656 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c65456d-1736-4e44-b02f-bdaad613572e None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:14.978933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c65456d-1736-4e44-b02f-bdaad613572e None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:14.979205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c65456d-1736-4e44-b02f-bdaad613572e None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:14.979919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c65456d-1736-4e44-b02f-bdaad613572e None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:14.980221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c65456d-1736-4e44-b02f-bdaad613572e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:14.981678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c65456d-1736-4e44-b02f-bdaad613572e None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:14.999408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 556/1097] 10.222.0.42 () {56 vars in 1133 bytes} [Sun Sep 4 00:48:14 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:15.005681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c65456d-1736-4e44-b02f-bdaad613572e None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:15.009870 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5248350-4fd8-456f-b5f5-8a2f647fa40e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:15.011296 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5248350-4fd8-456f-b5f5-8a2f647fa40e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:15.037842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:15.057171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:15.103971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c65456d-1736-4e44-b02f-bdaad613572e None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:15.111327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 542/1098] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:48:14 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 04 00:48:15.117174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5248350-4fd8-456f-b5f5-8a2f647fa40e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:15.139686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f972005-adb6-4572-87c3-6673128f3b99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:15.140288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f972005-adb6-4572-87c3-6673128f3b99 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:15.140573 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f972005-adb6-4572-87c3-6673128f3b99 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:15.140862 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f972005-adb6-4572-87c3-6673128f3b99 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:15.206612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f972005-adb6-4572-87c3-6673128f3b99 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:15.208044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f972005-adb6-4572-87c3-6673128f3b99 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:15.215657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5248350-4fd8-456f-b5f5-8a2f647fa40e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:15.240733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:15.248991 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a928902-0810-4bcf-b1cc-b7ca725877b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:15.249638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a928902-0810-4bcf-b1cc-b7ca725877b5 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:15.249915 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a928902-0810-4bcf-b1cc-b7ca725877b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:15.250295 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a928902-0810-4bcf-b1cc-b7ca725877b5 None admin] PATH_INFO: `/v3/projects/a46e66ec57f04506a481aba0e2838c2c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:15.251397 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a928902-0810-4bcf-b1cc-b7ca725877b5 None admin] RBAC: Authorizing `identity:get_project(project_id=a46e66ec57f04506a481aba0e2838c2c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:15.252803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a928902-0810-4bcf-b1cc-b7ca725877b5 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:15.254436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 543/1099] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:48:15 2022] GET /identity/v3/projects/a46e66ec57f04506a481aba0e2838c2c => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:15.258569 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:15.260851 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a928902-0810-4bcf-b1cc-b7ca725877b5 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:15.307111 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 557/1100] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:48:15 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 04 00:48:15.313432 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f972005-adb6-4572-87c3-6673128f3b99 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:15.368354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a928902-0810-4bcf-b1cc-b7ca725877b5 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:15.394571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47ea708-b5a6-4803-9b9d-627321e9a1d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:15.395174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47ea708-b5a6-4803-9b9d-627321e9a1d8 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:15.395449 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47ea708-b5a6-4803-9b9d-627321e9a1d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:15.397469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47ea708-b5a6-4803-9b9d-627321e9a1d8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:15.397469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47ea708-b5a6-4803-9b9d-627321e9a1d8 None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=a46e66ec57f04506a481aba0e2838c2c, user.id=8781b422f340440a87cc89b8fb2d20dc) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:15.397469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47ea708-b5a6-4803-9b9d-627321e9a1d8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:15.398306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47ea708-b5a6-4803-9b9d-627321e9a1d8 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:15.404005 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 544/1101] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:48:15 2022] GET /identity/v3/role_assignments?user.id=8781b422f340440a87cc89b8fb2d20dc&scope.project.id=a46e66ec57f04506a481aba0e2838c2c&role.id=19c006412bec45dc8febd7cc86b9678f => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:15.414735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f972005-adb6-4572-87c3-6673128f3b99 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:15.452083 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f038a03e-dd12-4c22-aa08-8badf9b8b64f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:15.452732 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f038a03e-dd12-4c22-aa08-8badf9b8b64f None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:15.453007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f038a03e-dd12-4c22-aa08-8badf9b8b64f None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:15.453342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f038a03e-dd12-4c22-aa08-8badf9b8b64f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:15.464855 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f038a03e-dd12-4c22-aa08-8badf9b8b64f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:15.466379 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f038a03e-dd12-4c22-aa08-8badf9b8b64f None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:15.482488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-f038a03e-dd12-4c22-aa08-8badf9b8b64f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:48:15.482488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:15.482488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:15.482488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:15.482488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:15.482488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:15.482488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:15.482488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:15.482488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:15.482488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:15.482488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:15.482488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:15.482488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:48:15.482488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:48:15.482488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:48:15.482488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:48:15.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:15.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:15.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:15.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:15.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:15.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:15.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:15.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:15.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:15.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:15.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:15.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:15.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:15.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:15.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:15.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:15.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:48:15.484495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:48:15.484495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:48:15.484495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:48:15.484495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 558/1102] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:48:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:48:15.490247 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47ea708-b5a6-4803-9b9d-627321e9a1d8 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:15.551883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:15.595518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47ea708-b5a6-4803-9b9d-627321e9a1d8 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:15.629522 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379429fd-25c7-4d74-a23b-908f1d220664 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:15.630049 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379429fd-25c7-4d74-a23b-908f1d220664 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:15.630328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379429fd-25c7-4d74-a23b-908f1d220664 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:15.630598 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379429fd-25c7-4d74-a23b-908f1d220664 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:15.631246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379429fd-25c7-4d74-a23b-908f1d220664 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:15.631552 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379429fd-25c7-4d74-a23b-908f1d220664 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:15.632934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379429fd-25c7-4d74-a23b-908f1d220664 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:15.645184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 545/1103] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:48:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:15.650972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f038a03e-dd12-4c22-aa08-8badf9b8b64f None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:15.654784 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:15.753107 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f038a03e-dd12-4c22-aa08-8badf9b8b64f None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:15.786079 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda09c09-c060-413d-899e-25d81be38cc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:15.786712 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda09c09-c060-413d-899e-25d81be38cc7 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:15.786993 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda09c09-c060-413d-899e-25d81be38cc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:15.787267 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda09c09-c060-413d-899e-25d81be38cc7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:15.793843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda09c09-c060-413d-899e-25d81be38cc7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:15.795276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda09c09-c060-413d-899e-25d81be38cc7 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:15.807045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-eda09c09-c060-413d-899e-25d81be38cc7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:48:15.807045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:15.807045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:15.807045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:15.807045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:15.807045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:15.807045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:15.807045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:15.807045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:15.807045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:15.807045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:15.807045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:15.807045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:48:15.807045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:48:15.807045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:48:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:48:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:15.808773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:48:15.808773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:48:15.808773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:48:15.808773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:48:15.808773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:48:15.808773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:48:15.808773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:48:15.808773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:48:15.808773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 559/1104] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:48:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:48:15.814585 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379429fd-25c7-4d74-a23b-908f1d220664 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:15.912310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379429fd-25c7-4d74-a23b-908f1d220664 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:15.929821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:15.946760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa4ba2b-29e9-480b-a242-fe121937e8f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:15.947357 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa4ba2b-29e9-480b-a242-fe121937e8f4 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:15.947637 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa4ba2b-29e9-480b-a242-fe121937e8f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:15.947907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa4ba2b-29e9-480b-a242-fe121937e8f4 None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:15.948595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa4ba2b-29e9-480b-a242-fe121937e8f4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:15.948899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa4ba2b-29e9-480b-a242-fe121937e8f4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:15.950347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa4ba2b-29e9-480b-a242-fe121937e8f4 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:15.961564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 546/1105] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:48:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:15.967627 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda09c09-c060-413d-899e-25d81be38cc7 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:16.030721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:16.069105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda09c09-c060-413d-899e-25d81be38cc7 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:16.102342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb47f03-ccb0-4766-b8d3-b3a162783249 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:16.102944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb47f03-ccb0-4766-b8d3-b3a162783249 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:16.103222 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb47f03-ccb0-4766-b8d3-b3a162783249 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:16.103491 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb47f03-ccb0-4766-b8d3-b3a162783249 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:16.104203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb47f03-ccb0-4766-b8d3-b3a162783249 None admin] RBAC: Adding query filter params (role.id=ace99e88214440f19edb734e93392300, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=57c88f16c7df40adb917f49804fda21f) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:16.104510 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb47f03-ccb0-4766-b8d3-b3a162783249 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:16.105953 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb47f03-ccb0-4766-b8d3-b3a162783249 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:16.111809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 560/1106] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:48:15 2022] GET /identity/v3/role_assignments?user.id=57c88f16c7df40adb917f49804fda21f&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=ace99e88214440f19edb734e93392300 => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:48:17.334692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3c1e2a-dbc1-4cba-a686-ac0feeea9a92 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:17.335254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3c1e2a-dbc1-4cba-a686-ac0feeea9a92 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:17.335254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3c1e2a-dbc1-4cba-a686-ac0feeea9a92 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:17.336616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 547/1107] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:17.341625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527af0e7-faf9-4ad6-8790-1e5053359e58 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:17.342027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527af0e7-faf9-4ad6-8790-1e5053359e58 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:17.342175 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527af0e7-faf9-4ad6-8790-1e5053359e58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:17.360727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-527af0e7-faf9-4ad6-8790-1e5053359e58 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:17.361931 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-527af0e7-faf9-4ad6-8790-1e5053359e58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:17.427257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 561/1108] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:17.695314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3c1e2a-dbc1-4cba-a686-ac0feeea9a92 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:17.793912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3c1e2a-dbc1-4cba-a686-ac0feeea9a92 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:17.826227 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01ba11d-5eda-4fda-a08a-44be44fef35e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:17.826832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01ba11d-5eda-4fda-a08a-44be44fef35e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:17.827108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01ba11d-5eda-4fda-a08a-44be44fef35e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:17.827377 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01ba11d-5eda-4fda-a08a-44be44fef35e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:17.828521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01ba11d-5eda-4fda-a08a-44be44fef35e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:17.829947 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01ba11d-5eda-4fda-a08a-44be44fef35e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:17.831466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 548/1109] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:48:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:17.837119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527af0e7-faf9-4ad6-8790-1e5053359e58 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:17.893891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:17.934649 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527af0e7-faf9-4ad6-8790-1e5053359e58 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:17.967423 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff5713e-78be-452a-ade4-ead1ee0ca109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:17.967950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff5713e-78be-452a-ade4-ead1ee0ca109 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:17.968230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff5713e-78be-452a-ade4-ead1ee0ca109 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:17.968502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff5713e-78be-452a-ade4-ead1ee0ca109 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:17.969838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff5713e-78be-452a-ade4-ead1ee0ca109 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:17.971255 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff5713e-78be-452a-ade4-ead1ee0ca109 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:18.071065 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 562/1110] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:48:17 2022] POST /identity/v3/users => generated 286 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:48:20.053871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d6f289-9f13-40ca-94b6-2c774518a84e None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:20.054464 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d6f289-9f13-40ca-94b6-2c774518a84e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:20.054464 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d6f289-9f13-40ca-94b6-2c774518a84e None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:20.055729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 549/1111] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:20.060902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e07f01d-95f8-4748-bc8c-c74bbdea8865 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:20.061375 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e07f01d-95f8-4748-bc8c-c74bbdea8865 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:20.061501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e07f01d-95f8-4748-bc8c-c74bbdea8865 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:20.080479 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-6e07f01d-95f8-4748-bc8c-c74bbdea8865 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:20.081688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-6e07f01d-95f8-4748-bc8c-c74bbdea8865 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:20.146223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 563/1112] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:20.418253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d6f289-9f13-40ca-94b6-2c774518a84e None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:20.466318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:20.521975 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d6f289-9f13-40ca-94b6-2c774518a84e None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:20.554391 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f888f17-9bfc-4881-9747-8ec13e03e101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:20.555008 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f888f17-9bfc-4881-9747-8ec13e03e101 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:20.555287 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f888f17-9bfc-4881-9747-8ec13e03e101 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:20.555559 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f888f17-9bfc-4881-9747-8ec13e03e101 None admin] PATH_INFO: `/v3/roles/c599b217579e42a49c657c642ea0648f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:20.556922 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f888f17-9bfc-4881-9747-8ec13e03e101 None admin] RBAC: Authorizing `identity:get_role(role_id=c599b217579e42a49c657c642ea0648f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:20.558298 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f888f17-9bfc-4881-9747-8ec13e03e101 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:20.559603 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 550/1113] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:48:20 2022] GET /identity/v3/roles/c599b217579e42a49c657c642ea0648f => generated 223 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:20.566146 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e07f01d-95f8-4748-bc8c-c74bbdea8865 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:20.610286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:20.661968 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e07f01d-95f8-4748-bc8c-c74bbdea8865 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:20.697280 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e505b7-8254-4c85-a21b-4ec6993103a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:20.697836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e505b7-8254-4c85-a21b-4ec6993103a7 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:20.698114 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e505b7-8254-4c85-a21b-4ec6993103a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:20.698386 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e505b7-8254-4c85-a21b-4ec6993103a7 None admin] PATH_INFO: `/v3/users/e7ab8c89081c4794a56891567f335f0e` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:20.711786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e505b7-8254-4c85-a21b-4ec6993103a7 None admin] RBAC: Authorizing `identity:get_user(user_id=e7ab8c89081c4794a56891567f335f0e)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:20.713246 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e505b7-8254-4c85-a21b-4ec6993103a7 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:20.714819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 564/1114] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:48:20 2022] GET /identity/v3/users/e7ab8c89081c4794a56891567f335f0e => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:20.721122 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f888f17-9bfc-4881-9747-8ec13e03e101 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:20.817528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f888f17-9bfc-4881-9747-8ec13e03e101 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:20.830921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:20.849587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11f6d84-7a17-4448-8ac6-f29fd4593594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:20.850168 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f6d84-7a17-4448-8ac6-f29fd4593594 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:20.850443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f6d84-7a17-4448-8ac6-f29fd4593594 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:20.850713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11f6d84-7a17-4448-8ac6-f29fd4593594 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:20.915077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11f6d84-7a17-4448-8ac6-f29fd4593594 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:20.916487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11f6d84-7a17-4448-8ac6-f29fd4593594 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:21.009409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 551/1115] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:48:20 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 04 00:48:21.015207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e505b7-8254-4c85-a21b-4ec6993103a7 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:21.112231 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e505b7-8254-4c85-a21b-4ec6993103a7 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:21.113748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:21.145095 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c5e401-f3fe-4c51-b055-ee1ef0195ce1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:21.145719 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c5e401-f3fe-4c51-b055-ee1ef0195ce1 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:21.145992 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c5e401-f3fe-4c51-b055-ee1ef0195ce1 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:21.146259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c5e401-f3fe-4c51-b055-ee1ef0195ce1 None admin] PATH_INFO: `/v3/projects/a46e66ec57f04506a481aba0e2838c2c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:21.147347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c5e401-f3fe-4c51-b055-ee1ef0195ce1 None admin] RBAC: Authorizing `identity:get_project(project_id=a46e66ec57f04506a481aba0e2838c2c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:21.148750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c5e401-f3fe-4c51-b055-ee1ef0195ce1 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:21.150370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 565/1116] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:48:21 2022] GET /identity/v3/projects/a46e66ec57f04506a481aba0e2838c2c => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:21.157231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11f6d84-7a17-4448-8ac6-f29fd4593594 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:21.171784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:21.253538 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11f6d84-7a17-4448-8ac6-f29fd4593594 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:21.287403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f3a3b8-c8df-40b9-9d23-bc0a9e6aca79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:21.287947 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f3a3b8-c8df-40b9-9d23-bc0a9e6aca79 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:21.288230 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f3a3b8-c8df-40b9-9d23-bc0a9e6aca79 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:21.288505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f3a3b8-c8df-40b9-9d23-bc0a9e6aca79 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:21.289258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f3a3b8-c8df-40b9-9d23-bc0a9e6aca79 None admin] RBAC: Adding query filter params (role.id=c599b217579e42a49c657c642ea0648f, scope.project.id=a46e66ec57f04506a481aba0e2838c2c, user.id=e7ab8c89081c4794a56891567f335f0e) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:21.289569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f3a3b8-c8df-40b9-9d23-bc0a9e6aca79 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:21.291020 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f3a3b8-c8df-40b9-9d23-bc0a9e6aca79 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:21.296192 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 552/1117] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:48:21 2022] GET /identity/v3/role_assignments?user.id=e7ab8c89081c4794a56891567f335f0e&scope.project.id=a46e66ec57f04506a481aba0e2838c2c&role.id=c599b217579e42a49c657c642ea0648f => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:23.196000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973ebcd0-8c8f-4f5a-b644-dc3658ee8517 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:23.196563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973ebcd0-8c8f-4f5a-b644-dc3658ee8517 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:23.196563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973ebcd0-8c8f-4f5a-b644-dc3658ee8517 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:23.197982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 566/1118] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:23.203693 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774b5da3-acf4-45f4-a3b9-9e4668dd66de None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:23.204159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774b5da3-acf4-45f4-a3b9-9e4668dd66de None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:23.204302 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774b5da3-acf4-45f4-a3b9-9e4668dd66de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:23.222024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-774b5da3-acf4-45f4-a3b9-9e4668dd66de None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:23.223193 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-774b5da3-acf4-45f4-a3b9-9e4668dd66de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:23.288329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 553/1119] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:48:23.557998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973ebcd0-8c8f-4f5a-b644-dc3658ee8517 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:23.656724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973ebcd0-8c8f-4f5a-b644-dc3658ee8517 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:23.689784 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a43ee2a-afc2-45de-bc18-7a5ec66c3404 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:23.690408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a43ee2a-afc2-45de-bc18-7a5ec66c3404 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:23.690688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a43ee2a-afc2-45de-bc18-7a5ec66c3404 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:23.690964 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a43ee2a-afc2-45de-bc18-7a5ec66c3404 None admin] PATH_INFO: `/v3/roles/c599b217579e42a49c657c642ea0648f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:23.692349 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a43ee2a-afc2-45de-bc18-7a5ec66c3404 None admin] RBAC: Authorizing `identity:get_role(role_id=c599b217579e42a49c657c642ea0648f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:23.693741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a43ee2a-afc2-45de-bc18-7a5ec66c3404 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:23.695040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 567/1120] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:48:23 2022] GET /identity/v3/roles/c599b217579e42a49c657c642ea0648f => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:23.700733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774b5da3-acf4-45f4-a3b9-9e4668dd66de None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:23.710708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:23.798615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774b5da3-acf4-45f4-a3b9-9e4668dd66de None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:23.830591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fb79ec-e5f6-4887-83e7-7da2f5553cff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:23.831223 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fb79ec-e5f6-4887-83e7-7da2f5553cff None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:23.831504 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fb79ec-e5f6-4887-83e7-7da2f5553cff None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:23.831778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fb79ec-e5f6-4887-83e7-7da2f5553cff None admin] PATH_INFO: `/v3/users/e7ab8c89081c4794a56891567f335f0e` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:23.832908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6fb79ec-e5f6-4887-83e7-7da2f5553cff None admin] RBAC: Authorizing `identity:get_user(user_id=e7ab8c89081c4794a56891567f335f0e)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:23.834366 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6fb79ec-e5f6-4887-83e7-7da2f5553cff None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:23.835937 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 554/1121] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:48:23 2022] GET /identity/v3/users/e7ab8c89081c4794a56891567f335f0e => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:23.841636 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a43ee2a-afc2-45de-bc18-7a5ec66c3404 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:23.881419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:23.939220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a43ee2a-afc2-45de-bc18-7a5ec66c3404 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:23.972640 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a352e5f0-830a-42c6-ba61-befdaedb25ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:23.973284 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a352e5f0-830a-42c6-ba61-befdaedb25ca None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:23.973567 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a352e5f0-830a-42c6-ba61-befdaedb25ca None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:23.973841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a352e5f0-830a-42c6-ba61-befdaedb25ca None admin] PATH_INFO: `/v3/projects/a46e66ec57f04506a481aba0e2838c2c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:23.974955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a352e5f0-830a-42c6-ba61-befdaedb25ca None admin] RBAC: Authorizing `identity:get_project(project_id=a46e66ec57f04506a481aba0e2838c2c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:23.976366 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a352e5f0-830a-42c6-ba61-befdaedb25ca None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:23.977994 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 568/1122] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:48:23 2022] GET /identity/v3/projects/a46e66ec57f04506a481aba0e2838c2c => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:23.983847 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fb79ec-e5f6-4887-83e7-7da2f5553cff None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:24.059707 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:24.082057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fb79ec-e5f6-4887-83e7-7da2f5553cff None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:24.115765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cb7d00-6b03-4ac8-b7f4-3f3debfb1d71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:24.116411 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cb7d00-6b03-4ac8-b7f4-3f3debfb1d71 None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:24.116691 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cb7d00-6b03-4ac8-b7f4-3f3debfb1d71 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:24.116967 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cb7d00-6b03-4ac8-b7f4-3f3debfb1d71 None admin] PATH_INFO: `/v3/projects/a46e66ec57f04506a481aba0e2838c2c/users/e7ab8c89081c4794a56891567f335f0e/roles/c599b217579e42a49c657c642ea0648f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:24.119126 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cb7d00-6b03-4ac8-b7f4-3f3debfb1d71 None admin] RBAC: Authorizing `identity:create_grant(project_id=a46e66ec57f04506a481aba0e2838c2c, user_id=e7ab8c89081c4794a56891567f335f0e, role_id=c599b217579e42a49c657c642ea0648f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:24.120546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cb7d00-6b03-4ac8-b7f4-3f3debfb1d71 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:24.167679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 555/1123] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:48:23 2022] PUT /identity/v3/projects/a46e66ec57f04506a481aba0e2838c2c/users/e7ab8c89081c4794a56891567f335f0e/roles/c599b217579e42a49c657c642ea0648f => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:48:26.105279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e7d4fc-3c33-411b-abd6-1618e5608a82 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:26.105857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e7d4fc-3c33-411b-abd6-1618e5608a82 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:26.105857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e7d4fc-3c33-411b-abd6-1618e5608a82 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:26.107186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 569/1124] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:26.112891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a1e367-683b-487c-8bb6-ab8d78b6e0b8 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:26.113354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a1e367-683b-487c-8bb6-ab8d78b6e0b8 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:26.113455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a1e367-683b-487c-8bb6-ab8d78b6e0b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:26.131433 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-44a1e367-683b-487c-8bb6-ab8d78b6e0b8 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:26.132592 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-44a1e367-683b-487c-8bb6-ab8d78b6e0b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:26.196460 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 556/1125] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:26.468555 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e7d4fc-3c33-411b-abd6-1618e5608a82 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:26.567789 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e7d4fc-3c33-411b-abd6-1618e5608a82 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:26.574833 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:26.601309 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2f401e-9699-4874-b20e-6a34c5a377c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:26.601930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2f401e-9699-4874-b20e-6a34c5a377c9 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:26.602208 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2f401e-9699-4874-b20e-6a34c5a377c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:26.602478 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2f401e-9699-4874-b20e-6a34c5a377c9 None admin] PATH_INFO: `/v3/roles/c599b217579e42a49c657c642ea0648f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:26.603856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2f401e-9699-4874-b20e-6a34c5a377c9 None admin] RBAC: Authorizing `identity:get_role(role_id=c599b217579e42a49c657c642ea0648f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:26.605253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2f401e-9699-4874-b20e-6a34c5a377c9 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:26.606553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 570/1126] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:48:26 2022] GET /identity/v3/roles/c599b217579e42a49c657c642ea0648f => generated 223 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:26.612256 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a1e367-683b-487c-8bb6-ab8d78b6e0b8 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:26.675323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:26.716726 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a1e367-683b-487c-8bb6-ab8d78b6e0b8 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:26.749243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3b0185-4904-41c7-b9ee-9043cafc434d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:26.750398 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3b0185-4904-41c7-b9ee-9043cafc434d None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:26.750678 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3b0185-4904-41c7-b9ee-9043cafc434d None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:26.750952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3b0185-4904-41c7-b9ee-9043cafc434d None admin] PATH_INFO: `/v3/users/e7ab8c89081c4794a56891567f335f0e` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:26.752076 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3b0185-4904-41c7-b9ee-9043cafc434d None admin] RBAC: Authorizing `identity:get_user(user_id=e7ab8c89081c4794a56891567f335f0e)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:26.753553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3b0185-4904-41c7-b9ee-9043cafc434d None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:26.755148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 557/1127] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:48:26 2022] GET /identity/v3/users/e7ab8c89081c4794a56891567f335f0e => generated 286 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:26.760828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2f401e-9699-4874-b20e-6a34c5a377c9 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:26.857752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2f401e-9699-4874-b20e-6a34c5a377c9 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:26.891044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344dcf39-8a2d-48ea-bba7-f960902dd649 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:26.891636 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344dcf39-8a2d-48ea-bba7-f960902dd649 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:26.891912 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344dcf39-8a2d-48ea-bba7-f960902dd649 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:26.892182 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344dcf39-8a2d-48ea-bba7-f960902dd649 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:26.932504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:26.958232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344dcf39-8a2d-48ea-bba7-f960902dd649 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:26.959686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344dcf39-8a2d-48ea-bba7-f960902dd649 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:27.053616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 571/1128] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:48:26 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 04 00:48:27.059484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3b0185-4904-41c7-b9ee-9043cafc434d None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:27.156132 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3b0185-4904-41c7-b9ee-9043cafc434d None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:27.188972 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03110bd6-a6a7-457c-bd71-12938071b14a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:27.189611 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03110bd6-a6a7-457c-bd71-12938071b14a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:27.189891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03110bd6-a6a7-457c-bd71-12938071b14a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:27.190163 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03110bd6-a6a7-457c-bd71-12938071b14a None admin] PATH_INFO: `/v3/projects/a46e66ec57f04506a481aba0e2838c2c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:27.191270 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03110bd6-a6a7-457c-bd71-12938071b14a None admin] RBAC: Authorizing `identity:get_project(project_id=a46e66ec57f04506a481aba0e2838c2c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:27.192682 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03110bd6-a6a7-457c-bd71-12938071b14a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:27.194310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 558/1129] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:48:27 2022] GET /identity/v3/projects/a46e66ec57f04506a481aba0e2838c2c => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:27.200391 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344dcf39-8a2d-48ea-bba7-f960902dd649 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:27.272904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:27.297169 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344dcf39-8a2d-48ea-bba7-f960902dd649 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:27.330111 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31a8f11-5c41-4b23-a653-30db358db3f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:27.330693 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31a8f11-5c41-4b23-a653-30db358db3f2 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:27.330966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31a8f11-5c41-4b23-a653-30db358db3f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:27.331231 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31a8f11-5c41-4b23-a653-30db358db3f2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:27.331933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31a8f11-5c41-4b23-a653-30db358db3f2 None admin] RBAC: Adding query filter params (role.id=c599b217579e42a49c657c642ea0648f, scope.project.id=a46e66ec57f04506a481aba0e2838c2c, user.id=e7ab8c89081c4794a56891567f335f0e) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:27.332234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31a8f11-5c41-4b23-a653-30db358db3f2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:27.333671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31a8f11-5c41-4b23-a653-30db358db3f2 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:27.339217 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 572/1130] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:48:27 2022] GET /identity/v3/role_assignments?user.id=e7ab8c89081c4794a56891567f335f0e&scope.project.id=a46e66ec57f04506a481aba0e2838c2c&role.id=c599b217579e42a49c657c642ea0648f => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:29.250282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55a26ec-972b-4e0a-adfc-c31bb284ac68 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:29.250839 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55a26ec-972b-4e0a-adfc-c31bb284ac68 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:29.250839 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55a26ec-972b-4e0a-adfc-c31bb284ac68 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:29.252140 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 559/1131] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:29.257900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8039012-fff2-4afa-9d6a-d2cb0cf35063 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:29.258279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8039012-fff2-4afa-9d6a-d2cb0cf35063 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:29.258443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8039012-fff2-4afa-9d6a-d2cb0cf35063 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:29.276322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-d8039012-fff2-4afa-9d6a-d2cb0cf35063 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:29.277526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d8039012-fff2-4afa-9d6a-d2cb0cf35063 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:29.341631 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 573/1132] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:29.610101 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55a26ec-972b-4e0a-adfc-c31bb284ac68 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:29.613008 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:29.710611 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55a26ec-972b-4e0a-adfc-c31bb284ac68 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:29.742622 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12522a23-b4d7-4ca2-a34c-0544c53afadd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:29.743230 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12522a23-b4d7-4ca2-a34c-0544c53afadd None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:29.743505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12522a23-b4d7-4ca2-a34c-0544c53afadd None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:29.743773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12522a23-b4d7-4ca2-a34c-0544c53afadd None admin] PATH_INFO: `/v3/domains/default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:29.744855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12522a23-b4d7-4ca2-a34c-0544c53afadd None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:29.746292 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12522a23-b4d7-4ca2-a34c-0544c53afadd None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:29.747825 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 560/1133] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:48:29 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:29.754649 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8039012-fff2-4afa-9d6a-d2cb0cf35063 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:29.819225 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:29.851194 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8039012-fff2-4afa-9d6a-d2cb0cf35063 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:29.884643 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc646c4c-f2ed-4f31-8497-57df5bfae613 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:29.885209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc646c4c-f2ed-4f31-8497-57df5bfae613 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:29.885484 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc646c4c-f2ed-4f31-8497-57df5bfae613 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:29.885751 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc646c4c-f2ed-4f31-8497-57df5bfae613 None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:29.886386 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc646c4c-f2ed-4f31-8497-57df5bfae613 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:29.887798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc646c4c-f2ed-4f31-8497-57df5bfae613 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:29.951971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 574/1134] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 4 00:48:29 2022] POST /identity/v3/projects => generated 311 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:48:31.854263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e01ca9-35c2-40b0-a104-22123dcea9fa None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:31.854780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e01ca9-35c2-40b0-a104-22123dcea9fa None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:31.854890 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e01ca9-35c2-40b0-a104-22123dcea9fa None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:31.856160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 561/1135] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:31.861377 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2709d95e-041f-46da-9e99-b51bf9810933 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:31.861761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2709d95e-041f-46da-9e99-b51bf9810933 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:31.861924 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2709d95e-041f-46da-9e99-b51bf9810933 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:31.880491 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-2709d95e-041f-46da-9e99-b51bf9810933 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:31.881687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-2709d95e-041f-46da-9e99-b51bf9810933 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:31.946755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 575/1136] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:32.215908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e01ca9-35c2-40b0-a104-22123dcea9fa None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:32.284713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:32.315971 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e01ca9-35c2-40b0-a104-22123dcea9fa None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:32.348799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938ae104-5318-49bf-a2d4-e798e0f4e723 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:32.349445 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938ae104-5318-49bf-a2d4-e798e0f4e723 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:32.349724 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938ae104-5318-49bf-a2d4-e798e0f4e723 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:32.349995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938ae104-5318-49bf-a2d4-e798e0f4e723 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:32.351103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938ae104-5318-49bf-a2d4-e798e0f4e723 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:32.352519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938ae104-5318-49bf-a2d4-e798e0f4e723 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:32.354094 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 562/1137] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:48:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:32.360405 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2709d95e-041f-46da-9e99-b51bf9810933 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:32.408674 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:32.458041 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2709d95e-041f-46da-9e99-b51bf9810933 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:32.492463 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae567de4-f89e-4d13-9865-25f55d12a45c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:32.492996 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae567de4-f89e-4d13-9865-25f55d12a45c None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:32.493304 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae567de4-f89e-4d13-9865-25f55d12a45c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:32.493574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae567de4-f89e-4d13-9865-25f55d12a45c None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:32.494208 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae567de4-f89e-4d13-9865-25f55d12a45c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:32.495619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae567de4-f89e-4d13-9865-25f55d12a45c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:32.598371 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 576/1138] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:48:32 2022] POST /identity/v3/users => generated 286 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:48:34.537487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0aeb08d-41a0-4304-9d11-fd873f3b2aa1 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:34.537969 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0aeb08d-41a0-4304-9d11-fd873f3b2aa1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:34.538052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0aeb08d-41a0-4304-9d11-fd873f3b2aa1 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:34.539362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 563/1139] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:34.544616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fea85f-818f-42c1-9f99-a4239886bb2f None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:34.545012 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fea85f-818f-42c1-9f99-a4239886bb2f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:34.545203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fea85f-818f-42c1-9f99-a4239886bb2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:34.564099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-52fea85f-818f-42c1-9f99-a4239886bb2f None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:34.565311 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-52fea85f-818f-42c1-9f99-a4239886bb2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:34.629140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 577/1140] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:34.898426 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0aeb08d-41a0-4304-9d11-fd873f3b2aa1 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:34.996250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0aeb08d-41a0-4304-9d11-fd873f3b2aa1 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:35.029918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafa8116-50fd-46ee-bff1-3ffd5e6cd1d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:35.030549 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafa8116-50fd-46ee-bff1-3ffd5e6cd1d9 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:35.030825 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafa8116-50fd-46ee-bff1-3ffd5e6cd1d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:35.031252 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafa8116-50fd-46ee-bff1-3ffd5e6cd1d9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:35.037761 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:35.041522 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafa8116-50fd-46ee-bff1-3ffd5e6cd1d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:48:35.046238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-cafa8116-50fd-46ee-bff1-3ffd5e6cd1d9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:48:35.046238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:35.046238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:35.046238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:35.046238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:35.046238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:35.046238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:35.046238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:35.046238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:35.046238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:35.046238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:35.046238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:35.046238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:48:35.046238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:48:35.046238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:48:35.046238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:48:35.047574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:48:35.047574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:48:35.047574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:48:35.047574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:48:35.047574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:35.047574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:35.047574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:35.047574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:35.047574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:35.047574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:35.047574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:35.047574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:35.047574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:35.047574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:35.047574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:35.047574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:35.048738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:35.048738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:35.048738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:35.048738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:35.048738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:48:35.048738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:48:35.048738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:48:35.048738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:48:35.048738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:48:35.048738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:48:35.048738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:48:35.048738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:35.048738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 564/1141] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:48:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:48:35.053308 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fea85f-818f-42c1-9f99-a4239886bb2f None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:35.087145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:35.150237 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fea85f-818f-42c1-9f99-a4239886bb2f None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:35.182095 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475ea637-e0a8-42f9-a287-88ee3e1a7070 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:35.182686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475ea637-e0a8-42f9-a287-88ee3e1a7070 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:35.182961 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475ea637-e0a8-42f9-a287-88ee3e1a7070 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:35.183228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475ea637-e0a8-42f9-a287-88ee3e1a7070 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:35.183883 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475ea637-e0a8-42f9-a287-88ee3e1a7070 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:35.184181 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475ea637-e0a8-42f9-a287-88ee3e1a7070 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:35.185569 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475ea637-e0a8-42f9-a287-88ee3e1a7070 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:35.193794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 578/1142] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:48:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:35.199457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafa8116-50fd-46ee-bff1-3ffd5e6cd1d9 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:35.298234 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafa8116-50fd-46ee-bff1-3ffd5e6cd1d9 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:35.331161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2c81e7-b204-4ed6-9828-4effcaa4aa02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:35.331779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2c81e7-b204-4ed6-9828-4effcaa4aa02 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:35.332057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2c81e7-b204-4ed6-9828-4effcaa4aa02 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:35.332329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2c81e7-b204-4ed6-9828-4effcaa4aa02 None admin] PATH_INFO: `/v3/users/8ea38efcd6064825886f86095382f3c9` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:35.345044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2c81e7-b204-4ed6-9828-4effcaa4aa02 None admin] RBAC: Authorizing `identity:get_user(user_id=8ea38efcd6064825886f86095382f3c9)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:35.346504 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2c81e7-b204-4ed6-9828-4effcaa4aa02 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:35.348082 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 565/1143] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:48:35 2022] GET /identity/v3/users/8ea38efcd6064825886f86095382f3c9 => generated 286 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:35.354119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475ea637-e0a8-42f9-a287-88ee3e1a7070 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:35.449225 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:35.453127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475ea637-e0a8-42f9-a287-88ee3e1a7070 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:35.486366 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e71d63b-3aa8-4430-80d9-a6b91694ba7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:35.486965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e71d63b-3aa8-4430-80d9-a6b91694ba7f None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:35.487240 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e71d63b-3aa8-4430-80d9-a6b91694ba7f None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:35.487511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e71d63b-3aa8-4430-80d9-a6b91694ba7f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:35.551713 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e71d63b-3aa8-4430-80d9-a6b91694ba7f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:35.553165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e71d63b-3aa8-4430-80d9-a6b91694ba7f None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:35.645802 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:35.650127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 579/1144] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:48:35 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 04 00:48:35.656799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2c81e7-b204-4ed6-9828-4effcaa4aa02 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:35.711968 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:35.755591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2c81e7-b204-4ed6-9828-4effcaa4aa02 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:35.788366 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94b24a4-0bd4-427e-bae7-3ae80990236c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:35.788984 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94b24a4-0bd4-427e-bae7-3ae80990236c None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:35.789292 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94b24a4-0bd4-427e-bae7-3ae80990236c None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:35.789568 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94b24a4-0bd4-427e-bae7-3ae80990236c None admin] PATH_INFO: `/v3/projects/c3684b4e152f473082c5d88bbc372977` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:35.790726 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94b24a4-0bd4-427e-bae7-3ae80990236c None admin] RBAC: Authorizing `identity:get_project(project_id=c3684b4e152f473082c5d88bbc372977)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:35.792143 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94b24a4-0bd4-427e-bae7-3ae80990236c None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:35.793778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 566/1145] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:48:35 2022] GET /identity/v3/projects/c3684b4e152f473082c5d88bbc372977 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:35.800453 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e71d63b-3aa8-4430-80d9-a6b91694ba7f None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:35.897963 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e71d63b-3aa8-4430-80d9-a6b91694ba7f None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:35.931527 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f0b901-36cd-4d61-8059-073fcfca68a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:35.932121 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f0b901-36cd-4d61-8059-073fcfca68a2 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:35.932402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f0b901-36cd-4d61-8059-073fcfca68a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:35.932674 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f0b901-36cd-4d61-8059-073fcfca68a2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:35.933457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f0b901-36cd-4d61-8059-073fcfca68a2 None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=c3684b4e152f473082c5d88bbc372977, user.id=8ea38efcd6064825886f86095382f3c9) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:35.933767 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f0b901-36cd-4d61-8059-073fcfca68a2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:35.935182 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f0b901-36cd-4d61-8059-073fcfca68a2 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:35.940870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 580/1146] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:48:35 2022] GET /identity/v3/role_assignments?user.id=8ea38efcd6064825886f86095382f3c9&scope.project.id=c3684b4e152f473082c5d88bbc372977&role.id=19c006412bec45dc8febd7cc86b9678f => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:37.816314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25dc04b-9f6b-4412-bd91-2c20a678f1e7 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:37.816843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25dc04b-9f6b-4412-bd91-2c20a678f1e7 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:37.816843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25dc04b-9f6b-4412-bd91-2c20a678f1e7 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:37.818236 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 567/1147] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:37.823940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b1f72d-216a-4048-a7f9-cf83e94c794c None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:37.824451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b1f72d-216a-4048-a7f9-cf83e94c794c None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:37.824544 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b1f72d-216a-4048-a7f9-cf83e94c794c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:37.843830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-e7b1f72d-216a-4048-a7f9-cf83e94c794c None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:37.845247 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e7b1f72d-216a-4048-a7f9-cf83e94c794c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:37.902855 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:37.912741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 581/1148] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:38.179997 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25dc04b-9f6b-4412-bd91-2c20a678f1e7 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:38.280028 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25dc04b-9f6b-4412-bd91-2c20a678f1e7 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:38.303478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:38.314262 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19943eb4-c7f4-4f4e-9cbc-94d52cc62b81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:38.314882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19943eb4-c7f4-4f4e-9cbc-94d52cc62b81 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:38.315167 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19943eb4-c7f4-4f4e-9cbc-94d52cc62b81 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:38.315439 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19943eb4-c7f4-4f4e-9cbc-94d52cc62b81 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:38.325656 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19943eb4-c7f4-4f4e-9cbc-94d52cc62b81 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:48:38.330415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-19943eb4-c7f4-4f4e-9cbc-94d52cc62b81 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:48:38.330415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:38.330415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:38.330415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:38.330415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:38.330415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:38.330415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:38.330415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:38.330415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:38.330415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:38.330415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:38.330415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:38.330415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:48:38.330415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:48:38.330415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:48:38.330415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:48:38.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:48:38.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:48:38.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:48:38.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:48:38.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:38.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:38.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:38.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:38.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:38.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:38.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:38.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:38.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:38.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:38.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:38.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:38.332314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:38.332314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:38.332314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:38.332314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:38.332314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:48:38.332314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:48:38.332314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:48:38.332314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:48:38.332314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:48:38.332314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:48:38.332314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:48:38.332314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:38.332314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 568/1149] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:48:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:48:38.337481 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b1f72d-216a-4048-a7f9-cf83e94c794c None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:38.432842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b1f72d-216a-4048-a7f9-cf83e94c794c None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:38.471571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2224eb67-713c-46d4-8607-df4a5b7697f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:38.472463 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2224eb67-713c-46d4-8607-df4a5b7697f0 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:38.472883 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2224eb67-713c-46d4-8607-df4a5b7697f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:38.473337 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2224eb67-713c-46d4-8607-df4a5b7697f0 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:38.474353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2224eb67-713c-46d4-8607-df4a5b7697f0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:38.474820 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2224eb67-713c-46d4-8607-df4a5b7697f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:38.476968 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2224eb67-713c-46d4-8607-df4a5b7697f0 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:38.489458 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 582/1150] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:48:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:38.495302 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19943eb4-c7f4-4f4e-9cbc-94d52cc62b81 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:38.594856 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19943eb4-c7f4-4f4e-9cbc-94d52cc62b81 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:38.628128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2377d482-412e-45ce-993f-30673cc5e41e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:38.628744 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2377d482-412e-45ce-993f-30673cc5e41e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:38.629019 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2377d482-412e-45ce-993f-30673cc5e41e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:38.629320 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2377d482-412e-45ce-993f-30673cc5e41e None admin] PATH_INFO: `/v3/users/8ea38efcd6064825886f86095382f3c9` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:38.630421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2377d482-412e-45ce-993f-30673cc5e41e None admin] RBAC: Authorizing `identity:get_user(user_id=8ea38efcd6064825886f86095382f3c9)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:38.631872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2377d482-412e-45ce-993f-30673cc5e41e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:38.633462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 569/1151] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:48:38 2022] GET /identity/v3/users/8ea38efcd6064825886f86095382f3c9 => generated 286 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:38.639146 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2224eb67-713c-46d4-8607-df4a5b7697f0 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:38.692779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:38.736352 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2224eb67-713c-46d4-8607-df4a5b7697f0 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:38.771378 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e05ab5e-a05f-442a-b15f-68d6bc06e13a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:38.771993 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e05ab5e-a05f-442a-b15f-68d6bc06e13a None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:38.772270 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e05ab5e-a05f-442a-b15f-68d6bc06e13a None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:38.772539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e05ab5e-a05f-442a-b15f-68d6bc06e13a None admin] PATH_INFO: `/v3/projects/c3684b4e152f473082c5d88bbc372977` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:38.773663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e05ab5e-a05f-442a-b15f-68d6bc06e13a None admin] RBAC: Authorizing `identity:get_project(project_id=c3684b4e152f473082c5d88bbc372977)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:38.775089 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e05ab5e-a05f-442a-b15f-68d6bc06e13a None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:38.776927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 583/1152] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:48:38 2022] GET /identity/v3/projects/c3684b4e152f473082c5d88bbc372977 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:38.783209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2377d482-412e-45ce-993f-30673cc5e41e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:38.836893 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:38.882810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2377d482-412e-45ce-993f-30673cc5e41e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:38.915174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c338d35-9604-42e0-b52d-770e0192fa86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:38.915816 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c338d35-9604-42e0-b52d-770e0192fa86 None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:38.916100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c338d35-9604-42e0-b52d-770e0192fa86 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:38.916375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c338d35-9604-42e0-b52d-770e0192fa86 None admin] PATH_INFO: `/v3/projects/c3684b4e152f473082c5d88bbc372977/users/8ea38efcd6064825886f86095382f3c9/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:38.918283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c338d35-9604-42e0-b52d-770e0192fa86 None admin] RBAC: Authorizing `identity:create_grant(project_id=c3684b4e152f473082c5d88bbc372977, user_id=8ea38efcd6064825886f86095382f3c9, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:38.919705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c338d35-9604-42e0-b52d-770e0192fa86 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:38.984687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 570/1153] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:48:38 2022] PUT /identity/v3/projects/c3684b4e152f473082c5d88bbc372977/users/8ea38efcd6064825886f86095382f3c9/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:48:40.867483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15da9ff-eb83-4fd7-804f-76db58d4192d None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:40.868013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15da9ff-eb83-4fd7-804f-76db58d4192d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:40.868013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15da9ff-eb83-4fd7-804f-76db58d4192d None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:40.869380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 584/1154] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:40.875256 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d0cf58-c079-41d7-9c13-aa312efaaed2 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:40.875701 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d0cf58-c079-41d7-9c13-aa312efaaed2 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:40.875794 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d0cf58-c079-41d7-9c13-aa312efaaed2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:40.894302 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-b5d0cf58-c079-41d7-9c13-aa312efaaed2 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:40.895485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b5d0cf58-c079-41d7-9c13-aa312efaaed2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:40.959217 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 571/1155] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:41.231334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15da9ff-eb83-4fd7-804f-76db58d4192d None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:41.330345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15da9ff-eb83-4fd7-804f-76db58d4192d None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:41.351835 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:41.363748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9a7628-2f57-4bdd-9f1e-889ccba2d0e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:41.364361 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9a7628-2f57-4bdd-9f1e-889ccba2d0e8 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:41.364637 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9a7628-2f57-4bdd-9f1e-889ccba2d0e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:41.364904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9a7628-2f57-4bdd-9f1e-889ccba2d0e8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:41.375615 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab9a7628-2f57-4bdd-9f1e-889ccba2d0e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:48:41.380717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-ab9a7628-2f57-4bdd-9f1e-889ccba2d0e8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:48:41.380717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:41.380717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:41.380717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:41.380717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:41.380717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:41.380717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:41.380717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:41.380717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:41.380717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:41.380717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:41.380717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:41.380717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:48:41.380717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:48:41.380717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:48:41.380717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:48:41.381856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:48:41.381856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:48:41.381856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:48:41.381856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:48:41.381856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:41.381856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:41.381856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:41.381856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:41.381856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:41.381856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:41.381856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:41.381856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:41.381856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:41.381856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:41.381856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:41.381856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:41.382688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:41.382688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:41.382688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:41.382688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:41.382688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:48:41.382688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:48:41.382688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:48:41.382688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:48:41.382688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:48:41.382688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:48:41.382688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:48:41.382688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:48:41.382688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 585/1156] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:48:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:48:41.387907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d0cf58-c079-41d7-9c13-aa312efaaed2 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:41.429716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:41.486109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d0cf58-c079-41d7-9c13-aa312efaaed2 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:41.519975 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769252cc-ba55-4cfc-988a-c01db804208e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:41.520582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769252cc-ba55-4cfc-988a-c01db804208e None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:41.520864 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769252cc-ba55-4cfc-988a-c01db804208e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:41.521174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769252cc-ba55-4cfc-988a-c01db804208e None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:41.521844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769252cc-ba55-4cfc-988a-c01db804208e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:41.522151 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769252cc-ba55-4cfc-988a-c01db804208e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:41.523508 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769252cc-ba55-4cfc-988a-c01db804208e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:41.531612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 572/1157] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:48:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:41.537563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9a7628-2f57-4bdd-9f1e-889ccba2d0e8 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:41.633979 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9a7628-2f57-4bdd-9f1e-889ccba2d0e8 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:41.667221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23c0c90-42f1-475c-8e78-59cb1db8d9a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:41.667825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23c0c90-42f1-475c-8e78-59cb1db8d9a7 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:41.668099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23c0c90-42f1-475c-8e78-59cb1db8d9a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:41.668364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d23c0c90-42f1-475c-8e78-59cb1db8d9a7 None admin] PATH_INFO: `/v3/users/8ea38efcd6064825886f86095382f3c9` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:41.669477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23c0c90-42f1-475c-8e78-59cb1db8d9a7 None admin] RBAC: Authorizing `identity:get_user(user_id=8ea38efcd6064825886f86095382f3c9)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:41.671657 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d23c0c90-42f1-475c-8e78-59cb1db8d9a7 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:41.673198 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 586/1158] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:48:41 2022] GET /identity/v3/users/8ea38efcd6064825886f86095382f3c9 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:41.679012 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769252cc-ba55-4cfc-988a-c01db804208e None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:41.774884 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769252cc-ba55-4cfc-988a-c01db804208e None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:41.777064 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:41.808865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89dbcf1-9c90-4b7a-95f7-c010eea826e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:41.809486 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89dbcf1-9c90-4b7a-95f7-c010eea826e6 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:41.809768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89dbcf1-9c90-4b7a-95f7-c010eea826e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:41.810040 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89dbcf1-9c90-4b7a-95f7-c010eea826e6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:41.875065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89dbcf1-9c90-4b7a-95f7-c010eea826e6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:41.876476 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89dbcf1-9c90-4b7a-95f7-c010eea826e6 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:41.971240 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 573/1159] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:48:41 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 04 00:48:41.977224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23c0c90-42f1-475c-8e78-59cb1db8d9a7 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:42.026073 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:42.074419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d23c0c90-42f1-475c-8e78-59cb1db8d9a7 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:42.107586 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cc92f2-1b15-4048-84dd-3e0f75860a38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:42.108221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cc92f2-1b15-4048-84dd-3e0f75860a38 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:42.108542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cc92f2-1b15-4048-84dd-3e0f75860a38 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:42.108806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cc92f2-1b15-4048-84dd-3e0f75860a38 None admin] PATH_INFO: `/v3/projects/c3684b4e152f473082c5d88bbc372977` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:42.109934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cc92f2-1b15-4048-84dd-3e0f75860a38 None admin] RBAC: Authorizing `identity:get_project(project_id=c3684b4e152f473082c5d88bbc372977)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:42.111349 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cc92f2-1b15-4048-84dd-3e0f75860a38 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:42.112943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 587/1160] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:48:41 2022] GET /identity/v3/projects/c3684b4e152f473082c5d88bbc372977 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:42.118975 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89dbcf1-9c90-4b7a-95f7-c010eea826e6 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:42.122003 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:42.215741 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89dbcf1-9c90-4b7a-95f7-c010eea826e6 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:42.248481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28949374-db68-4e05-85e5-b92991a7ceb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:42.249089 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28949374-db68-4e05-85e5-b92991a7ceb3 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:42.249367 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28949374-db68-4e05-85e5-b92991a7ceb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:42.249637 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28949374-db68-4e05-85e5-b92991a7ceb3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:42.250349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28949374-db68-4e05-85e5-b92991a7ceb3 None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=c3684b4e152f473082c5d88bbc372977, user.id=8ea38efcd6064825886f86095382f3c9) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:42.250654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28949374-db68-4e05-85e5-b92991a7ceb3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:42.252048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28949374-db68-4e05-85e5-b92991a7ceb3 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:42.257581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 574/1161] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:48:42 2022] GET /identity/v3/role_assignments?user.id=8ea38efcd6064825886f86095382f3c9&scope.project.id=c3684b4e152f473082c5d88bbc372977&role.id=19c006412bec45dc8febd7cc86b9678f => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:44.195041 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b40ba8-5b12-48b1-b517-4cd5237dab01 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:44.195571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b40ba8-5b12-48b1-b517-4cd5237dab01 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:44.195571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b40ba8-5b12-48b1-b517-4cd5237dab01 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:44.196900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 588/1162] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:44.202567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cc0bc1-249c-4f27-aad0-bc3346db50ff None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:44.202964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cc0bc1-249c-4f27-aad0-bc3346db50ff None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:44.203150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cc0bc1-249c-4f27-aad0-bc3346db50ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:44.221564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-43cc0bc1-249c-4f27-aad0-bc3346db50ff None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:44.222758 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-43cc0bc1-249c-4f27-aad0-bc3346db50ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:44.283174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:44.288246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 575/1163] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:44.557532 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b40ba8-5b12-48b1-b517-4cd5237dab01 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:44.658099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b40ba8-5b12-48b1-b517-4cd5237dab01 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:44.674940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:44.691628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f2b8bf-d771-48c2-90a6-55a92868c95a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:44.692221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f2b8bf-d771-48c2-90a6-55a92868c95a None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:44.693808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f2b8bf-d771-48c2-90a6-55a92868c95a None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:44.693808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f2b8bf-d771-48c2-90a6-55a92868c95a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:44.759223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f2b8bf-d771-48c2-90a6-55a92868c95a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:44.760600 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f2b8bf-d771-48c2-90a6-55a92868c95a None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:44.859962 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 589/1164] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:48:44 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 04 00:48:44.865874 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cc0bc1-249c-4f27-aad0-bc3346db50ff None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:44.961877 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cc0bc1-249c-4f27-aad0-bc3346db50ff None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:44.994910 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9582f16-ed45-47cb-8c7e-a274f23f6f22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:44.995524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9582f16-ed45-47cb-8c7e-a274f23f6f22 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:44.995805 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9582f16-ed45-47cb-8c7e-a274f23f6f22 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:44.996077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9582f16-ed45-47cb-8c7e-a274f23f6f22 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:45.002943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9582f16-ed45-47cb-8c7e-a274f23f6f22 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:45.004364 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9582f16-ed45-47cb-8c7e-a274f23f6f22 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:45.015992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-b9582f16-ed45-47cb-8c7e-a274f23f6f22 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 04 00:48:45.015992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:45.015992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 04 00:48:45.015992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 04 00:48:45.015992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:48:45.015992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:48:45.015992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:48:45.015992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:48:45.015992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 04 00:48:45.015992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:45.015992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:48:45.015992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:45.015992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:45.015992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:45.015992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:45.015992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:45.015992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:45.017297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:45.017297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:45.017297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:45.017297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:45.017297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:45.017297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:45.017297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:48:45.017297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:48:45.017297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:48:45.017297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:48:45.017297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:45.017297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:45.017297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:45.017297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:45.017297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:45.017297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:45.018419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:45.018419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:45.018419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:45.018419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:45.018419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:45.018419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:45.018419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:45.018419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:45.018419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:45.018419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:45.018419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 04 00:48:45.018419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:48:45.018419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 04 00:48:45.018419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:48:45.018419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 576/1165] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 4 00:48:44 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:48:45.022917 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f2b8bf-d771-48c2-90a6-55a92868c95a None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:45.041181 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:45.124982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f2b8bf-d771-48c2-90a6-55a92868c95a None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:45.157474 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d2e046-9e99-4158-9bfd-6b14477d5eaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:45.158068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d2e046-9e99-4158-9bfd-6b14477d5eaa None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:45.158347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d2e046-9e99-4158-9bfd-6b14477d5eaa None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:45.158619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d2e046-9e99-4158-9bfd-6b14477d5eaa None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:45.159256 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d2e046-9e99-4158-9bfd-6b14477d5eaa None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:45.159559 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d2e046-9e99-4158-9bfd-6b14477d5eaa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:45.160951 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d2e046-9e99-4158-9bfd-6b14477d5eaa None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:45.167880 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 590/1166] 10.222.0.42 () {56 vars in 1115 bytes} [Sun Sep 4 00:48:45 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:47.056177 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec28b3b-7d8c-4bf5-9099-5eb0b05ea67c None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:47.056743 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec28b3b-7d8c-4bf5-9099-5eb0b05ea67c None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:47.056743 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec28b3b-7d8c-4bf5-9099-5eb0b05ea67c None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:47.058093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 577/1167] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:47.063990 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d509b453-abd8-4516-9c3d-d2435b114d7b None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:47.064405 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d509b453-abd8-4516-9c3d-d2435b114d7b None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:47.064542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d509b453-abd8-4516-9c3d-d2435b114d7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:47.082799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-d509b453-abd8-4516-9c3d-d2435b114d7b None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:47.083967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d509b453-abd8-4516-9c3d-d2435b114d7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:47.148911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 591/1168] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:47.419361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec28b3b-7d8c-4bf5-9099-5eb0b05ea67c None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:47.480279 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:47.525317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec28b3b-7d8c-4bf5-9099-5eb0b05ea67c None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:47.558540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36639a6a-0e90-4861-ac43-1fcfbd754880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:47.559186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36639a6a-0e90-4861-ac43-1fcfbd754880 None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:47.559472 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36639a6a-0e90-4861-ac43-1fcfbd754880 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:47.559747 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36639a6a-0e90-4861-ac43-1fcfbd754880 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:47.560407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36639a6a-0e90-4861-ac43-1fcfbd754880 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:47.562631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36639a6a-0e90-4861-ac43-1fcfbd754880 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:47.616367 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 578/1169] 10.222.0.42 () {60 vars in 1138 bytes} [Sun Sep 4 00:48:47 2022] POST /identity/v3/domains => generated 267 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:48:49.535120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2b5974-e289-4e47-8b2c-0e359cb2ab57 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:49.535686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2b5974-e289-4e47-8b2c-0e359cb2ab57 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:49.535686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f2b5974-e289-4e47-8b2c-0e359cb2ab57 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:49.537014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 592/1170] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:49.542055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eda1d0d-234f-4d61-8cc7-96f4822a1dce None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:49.542446 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eda1d0d-234f-4d61-8cc7-96f4822a1dce None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:49.542604 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eda1d0d-234f-4d61-8cc7-96f4822a1dce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:49.561168 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-2eda1d0d-234f-4d61-8cc7-96f4822a1dce None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:49.562343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2eda1d0d-234f-4d61-8cc7-96f4822a1dce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:49.632457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 579/1171] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:49.901434 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2b5974-e289-4e47-8b2c-0e359cb2ab57 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:49.916741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:50.002029 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f2b5974-e289-4e47-8b2c-0e359cb2ab57 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:50.034535 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df47799-479e-48dc-a7fc-3fac31d1c3b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:50.035148 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df47799-479e-48dc-a7fc-3fac31d1c3b2 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:50.035424 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df47799-479e-48dc-a7fc-3fac31d1c3b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:50.035694 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df47799-479e-48dc-a7fc-3fac31d1c3b2 None admin] PATH_INFO: `/v3/domains/7520705bfd7a4895925056bc796fb0d1` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:50.046181 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df47799-479e-48dc-a7fc-3fac31d1c3b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=7520705bfd7a4895925056bc796fb0d1)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:50.047716 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df47799-479e-48dc-a7fc-3fac31d1c3b2 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:50.049298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 593/1172] 10.222.0.42 () {56 vars in 1183 bytes} [Sun Sep 4 00:48:49 2022] GET /identity/v3/domains/7520705bfd7a4895925056bc796fb0d1 => generated 267 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:50.055074 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eda1d0d-234f-4d61-8cc7-96f4822a1dce None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:50.101932 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:50.151877 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eda1d0d-234f-4d61-8cc7-96f4822a1dce None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:50.183929 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0424c63-4f61-4f70-ad08-640a16f79d1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:50.184524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0424c63-4f61-4f70-ad08-640a16f79d1b None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:50.184806 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0424c63-4f61-4f70-ad08-640a16f79d1b None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:50.185106 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0424c63-4f61-4f70-ad08-640a16f79d1b None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:50.185748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0424c63-4f61-4f70-ad08-640a16f79d1b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:50.187964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0424c63-4f61-4f70-ad08-640a16f79d1b None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:50.255338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 580/1173] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 4 00:48:50 2022] POST /identity/v3/projects => generated 361 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:48:52.175943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77948096-ad7d-4bdd-8f80-22e5e118d807 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:52.176503 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77948096-ad7d-4bdd-8f80-22e5e118d807 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:52.176503 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77948096-ad7d-4bdd-8f80-22e5e118d807 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:52.177892 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 594/1174] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:52.182909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3098351f-6485-4804-8756-e4f2853ead94 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:52.183300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3098351f-6485-4804-8756-e4f2853ead94 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:52.183454 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3098351f-6485-4804-8756-e4f2853ead94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:52.202613 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-3098351f-6485-4804-8756-e4f2853ead94 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:52.203788 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-3098351f-6485-4804-8756-e4f2853ead94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:52.268546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 581/1175] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:52.536268 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77948096-ad7d-4bdd-8f80-22e5e118d807 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:52.606819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:52.637878 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77948096-ad7d-4bdd-8f80-22e5e118d807 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:52.671533 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1c22f7-225b-4b96-804e-9723230214ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:52.672152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1c22f7-225b-4b96-804e-9723230214ee None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:52.672431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1c22f7-225b-4b96-804e-9723230214ee None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:52.672704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1c22f7-225b-4b96-804e-9723230214ee None admin] PATH_INFO: `/v3/domains/7520705bfd7a4895925056bc796fb0d1` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:52.673831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1c22f7-225b-4b96-804e-9723230214ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=7520705bfd7a4895925056bc796fb0d1)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:52.675281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1c22f7-225b-4b96-804e-9723230214ee None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:52.676823 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 595/1176] 10.222.0.42 () {56 vars in 1183 bytes} [Sun Sep 4 00:48:52 2022] GET /identity/v3/domains/7520705bfd7a4895925056bc796fb0d1 => generated 267 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:52.682551 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3098351f-6485-4804-8756-e4f2853ead94 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:52.711934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:52.780747 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3098351f-6485-4804-8756-e4f2853ead94 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:52.813322 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c18407-8918-48da-a1cf-2b9a5648a1ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:52.813918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c18407-8918-48da-a1cf-2b9a5648a1ee None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:52.814194 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c18407-8918-48da-a1cf-2b9a5648a1ee None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:52.814465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c18407-8918-48da-a1cf-2b9a5648a1ee None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:52.815123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c18407-8918-48da-a1cf-2b9a5648a1ee None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:52.816531 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c18407-8918-48da-a1cf-2b9a5648a1ee None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:52.911941 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 582/1177] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:48:52 2022] POST /identity/v3/users => generated 311 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:48:54.849711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6e2072-eb93-4565-812b-7419af4c35e4 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:54.850268 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6e2072-eb93-4565-812b-7419af4c35e4 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:54.850268 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6e2072-eb93-4565-812b-7419af4c35e4 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:54.851564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 596/1178] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:54.856617 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f92364-a802-47c0-8841-1af3f41fbecc None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:54.857058 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f92364-a802-47c0-8841-1af3f41fbecc None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:54.857195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f92364-a802-47c0-8841-1af3f41fbecc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:54.876356 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-03f92364-a802-47c0-8841-1af3f41fbecc None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:54.877584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-03f92364-a802-47c0-8841-1af3f41fbecc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:54.946341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 583/1179] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:55.215053 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6e2072-eb93-4565-812b-7419af4c35e4 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:55.314544 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6e2072-eb93-4565-812b-7419af4c35e4 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:55.346766 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0523e9b8-d47c-47ff-a6ba-f297767379de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:55.347364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0523e9b8-d47c-47ff-a6ba-f297767379de None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:55.347639 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0523e9b8-d47c-47ff-a6ba-f297767379de None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:55.347911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0523e9b8-d47c-47ff-a6ba-f297767379de None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:55.355131 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:55.358723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0523e9b8-d47c-47ff-a6ba-f297767379de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:48:55.363445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-0523e9b8-d47c-47ff-a6ba-f297767379de None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:48:55.363445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:55.363445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:55.363445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:55.363445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:55.363445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:55.363445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:55.363445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:55.363445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:55.363445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:55.363445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:55.363445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:55.363445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:48:55.363445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:48:55.363445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:48:55.363445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:48:55.364593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:48:55.364593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:48:55.364593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:48:55.364593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:48:55.364593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:55.364593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:55.364593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:55.364593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:55.364593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:55.364593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:55.364593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:55.364593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:55.364593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:55.364593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:55.364593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:55.364593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:55.365744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:55.365744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:55.365744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:55.365744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:55.365744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:48:55.365744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:48:55.365744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:48:55.365744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:48:55.365744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:48:55.365744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:48:55.365744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:48:55.365744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:48:55.365744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 597/1180] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:48:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:48:55.370672 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f92364-a802-47c0-8841-1af3f41fbecc None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:55.388611 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:55.470240 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f92364-a802-47c0-8841-1af3f41fbecc None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:55.504772 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ddad32-e263-49d3-91d1-16ac2e57a889 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:55.505396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ddad32-e263-49d3-91d1-16ac2e57a889 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:55.505678 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ddad32-e263-49d3-91d1-16ac2e57a889 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:55.505951 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ddad32-e263-49d3-91d1-16ac2e57a889 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:55.506624 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ddad32-e263-49d3-91d1-16ac2e57a889 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:55.506930 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ddad32-e263-49d3-91d1-16ac2e57a889 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:55.508290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ddad32-e263-49d3-91d1-16ac2e57a889 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:55.516768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 584/1181] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:48:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:55.522819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0523e9b8-d47c-47ff-a6ba-f297767379de None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:55.620924 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0523e9b8-d47c-47ff-a6ba-f297767379de None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:55.654302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb3611e-0799-41b2-b60e-c3bec8cba4fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:55.655644 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb3611e-0799-41b2-b60e-c3bec8cba4fd None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:55.655926 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb3611e-0799-41b2-b60e-c3bec8cba4fd None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:55.656204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb3611e-0799-41b2-b60e-c3bec8cba4fd None admin] PATH_INFO: `/v3/users/2def349673b14842a3ee9d67226cc3e8` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:55.668789 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb3611e-0799-41b2-b60e-c3bec8cba4fd None admin] RBAC: Authorizing `identity:get_user(user_id=2def349673b14842a3ee9d67226cc3e8)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:55.670303 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb3611e-0799-41b2-b60e-c3bec8cba4fd None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:55.671902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 598/1182] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:48:55 2022] GET /identity/v3/users/2def349673b14842a3ee9d67226cc3e8 => generated 311 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:55.677790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ddad32-e263-49d3-91d1-16ac2e57a889 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:55.753875 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:55.775143 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ddad32-e263-49d3-91d1-16ac2e57a889 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:55.808168 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec36331-df05-4579-bada-3af982857fee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:55.808760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec36331-df05-4579-bada-3af982857fee None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:55.809037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec36331-df05-4579-bada-3af982857fee None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:55.809340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec36331-df05-4579-bada-3af982857fee None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:55.874093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec36331-df05-4579-bada-3af982857fee None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:55.875508 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec36331-df05-4579-bada-3af982857fee None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:55.948585 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:55.970026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 585/1183] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:48:55 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 04 00:48:55.976145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb3611e-0799-41b2-b60e-c3bec8cba4fd None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:56.033204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:56.073387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb3611e-0799-41b2-b60e-c3bec8cba4fd None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:56.106896 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c13968-10f1-4261-ab66-616f8d276c02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:56.107511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c13968-10f1-4261-ab66-616f8d276c02 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:56.107840 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c13968-10f1-4261-ab66-616f8d276c02 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:56.108104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c13968-10f1-4261-ab66-616f8d276c02 None admin] PATH_INFO: `/v3/projects/166bb2b33981456dba5551ecb4010e91` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:56.109234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c13968-10f1-4261-ab66-616f8d276c02 None admin] RBAC: Authorizing `identity:get_project(project_id=166bb2b33981456dba5551ecb4010e91)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:56.110642 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c13968-10f1-4261-ab66-616f8d276c02 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:56.112229 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 599/1184] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:48:55 2022] GET /identity/v3/projects/166bb2b33981456dba5551ecb4010e91 => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:56.118427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec36331-df05-4579-bada-3af982857fee None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:56.213792 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec36331-df05-4579-bada-3af982857fee None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:56.246643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939015c3-c54c-4c43-a1e0-b873ba3bfbf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:56.247234 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939015c3-c54c-4c43-a1e0-b873ba3bfbf0 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:56.247509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939015c3-c54c-4c43-a1e0-b873ba3bfbf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:56.247778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939015c3-c54c-4c43-a1e0-b873ba3bfbf0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:56.249576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939015c3-c54c-4c43-a1e0-b873ba3bfbf0 None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=166bb2b33981456dba5551ecb4010e91, user.id=2def349673b14842a3ee9d67226cc3e8) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:56.249895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939015c3-c54c-4c43-a1e0-b873ba3bfbf0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:56.251290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939015c3-c54c-4c43-a1e0-b873ba3bfbf0 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:56.256414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 586/1185] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:48:56 2022] GET /identity/v3/role_assignments?user.id=2def349673b14842a3ee9d67226cc3e8&scope.project.id=166bb2b33981456dba5551ecb4010e91&role.id=19c006412bec45dc8febd7cc86b9678f => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:58.212314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a122b292-2a84-48a6-bc26-20b5d7662e95 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:58.212873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a122b292-2a84-48a6-bc26-20b5d7662e95 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:58.212873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a122b292-2a84-48a6-bc26-20b5d7662e95 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:58.214225 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 600/1186] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:48:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:48:58.220068 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28472f10-b4f7-434a-9e9c-a2253e36c33a None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:58.220507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28472f10-b4f7-434a-9e9c-a2253e36c33a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:58.220608 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28472f10-b4f7-434a-9e9c-a2253e36c33a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:58.238939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-28472f10-b4f7-434a-9e9c-a2253e36c33a None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:48:58.240109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-28472f10-b4f7-434a-9e9c-a2253e36c33a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:48:58.273765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:58.304587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 587/1187] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:48:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:48:58.575985 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a122b292-2a84-48a6-bc26-20b5d7662e95 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:58.675926 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a122b292-2a84-48a6-bc26-20b5d7662e95 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:58.700247 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:58.709318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b95606c-5d75-4dbf-8079-120713f4cdb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:58.712122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b95606c-5d75-4dbf-8079-120713f4cdb3 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:58.712122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b95606c-5d75-4dbf-8079-120713f4cdb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:58.712122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b95606c-5d75-4dbf-8079-120713f4cdb3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:58.721263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b95606c-5d75-4dbf-8079-120713f4cdb3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:48:58.726708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-1b95606c-5d75-4dbf-8079-120713f4cdb3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:48:58.726708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:48:58.726708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:48:58.726708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:48:58.726708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:48:58.726708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:48:58.726708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:48:58.726708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:48:58.726708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:48:58.726708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:48:58.726708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:48:58.726708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:48:58.726708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:48:58.726708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:48:58.726708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:48:58.726708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:48:58.727801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:48:58.727801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:48:58.727801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:48:58.727801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:48:58.727801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:48:58.727801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:48:58.727801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:48:58.727801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:48:58.727801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:48:58.727801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:48:58.727801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:48:58.727801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:48:58.727801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:48:58.727801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:48:58.727801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:48:58.727801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:48:58.728908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:48:58.728908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:48:58.728908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:48:58.728908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:48:58.728908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:48:58.728908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:48:58.728908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:48:58.728908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:48:58.728908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:48:58.728908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:48:58.728908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:48:58.728908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:48:58.728908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 601/1188] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:48:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:48:58.733970 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28472f10-b4f7-434a-9e9c-a2253e36c33a None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:58.829637 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28472f10-b4f7-434a-9e9c-a2253e36c33a None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:58.865405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84198831-8795-4944-8d33-7c543fe8f23c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:58.866010 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84198831-8795-4944-8d33-7c543fe8f23c None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:58.866288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84198831-8795-4944-8d33-7c543fe8f23c None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:58.866559 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84198831-8795-4944-8d33-7c543fe8f23c None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:58.867222 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84198831-8795-4944-8d33-7c543fe8f23c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:48:58.867525 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84198831-8795-4944-8d33-7c543fe8f23c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:58.868868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84198831-8795-4944-8d33-7c543fe8f23c None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:58.877554 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 588/1189] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:48:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:58.883391 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b95606c-5d75-4dbf-8079-120713f4cdb3 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:58.995400 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b95606c-5d75-4dbf-8079-120713f4cdb3 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:59.029069 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55315f53-d285-46fe-80c0-7f143f8deb1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:59.029747 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55315f53-d285-46fe-80c0-7f143f8deb1f None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:59.030024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55315f53-d285-46fe-80c0-7f143f8deb1f None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:59.030297 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55315f53-d285-46fe-80c0-7f143f8deb1f None admin] PATH_INFO: `/v3/users/2def349673b14842a3ee9d67226cc3e8` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:59.031405 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55315f53-d285-46fe-80c0-7f143f8deb1f None admin] RBAC: Authorizing `identity:get_user(user_id=2def349673b14842a3ee9d67226cc3e8)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:59.032827 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55315f53-d285-46fe-80c0-7f143f8deb1f None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:59.034430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 602/1190] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:48:58 2022] GET /identity/v3/users/2def349673b14842a3ee9d67226cc3e8 => generated 311 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:59.040284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84198831-8795-4944-8d33-7c543fe8f23c None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:59.074056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:59.137894 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84198831-8795-4944-8d33-7c543fe8f23c None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:59.170752 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4725e0-1ed3-4ab6-9e40-819238387732 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:59.171377 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4725e0-1ed3-4ab6-9e40-819238387732 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:59.171655 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4725e0-1ed3-4ab6-9e40-819238387732 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:59.171927 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4725e0-1ed3-4ab6-9e40-819238387732 None admin] PATH_INFO: `/v3/projects/166bb2b33981456dba5551ecb4010e91` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:59.173036 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4725e0-1ed3-4ab6-9e40-819238387732 None admin] RBAC: Authorizing `identity:get_project(project_id=166bb2b33981456dba5551ecb4010e91)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:59.174477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4725e0-1ed3-4ab6-9e40-819238387732 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:59.176080 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 589/1191] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:48:59 2022] GET /identity/v3/projects/166bb2b33981456dba5551ecb4010e91 => generated 361 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:48:59.181907 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55315f53-d285-46fe-80c0-7f143f8deb1f None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:48:59.236334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:48:59.279035 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55315f53-d285-46fe-80c0-7f143f8deb1f None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:48:59.311954 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2967f61c-c3d8-4206-b2e7-89e7fa0013cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:48:59.312596 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2967f61c-c3d8-4206-b2e7-89e7fa0013cc None admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:48:59.312875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2967f61c-c3d8-4206-b2e7-89e7fa0013cc None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:48:59.313173 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2967f61c-c3d8-4206-b2e7-89e7fa0013cc None admin] PATH_INFO: `/v3/projects/166bb2b33981456dba5551ecb4010e91/users/2def349673b14842a3ee9d67226cc3e8/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:48:59.315018 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2967f61c-c3d8-4206-b2e7-89e7fa0013cc None admin] RBAC: Authorizing `identity:create_grant(project_id=166bb2b33981456dba5551ecb4010e91, user_id=2def349673b14842a3ee9d67226cc3e8, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:48:59.316425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2967f61c-c3d8-4206-b2e7-89e7fa0013cc None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:48:59.401013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 603/1192] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:48:59 2022] PUT /identity/v3/projects/166bb2b33981456dba5551ecb4010e91/users/2def349673b14842a3ee9d67226cc3e8/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:49:01.300460 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6e577d-da57-4a6a-8d3e-b1129653c17e None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:49:01.301022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6e577d-da57-4a6a-8d3e-b1129653c17e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:49:01.301022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6e577d-da57-4a6a-8d3e-b1129653c17e None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:49:01.302457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 590/1193] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:49:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:49:01.307521 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816bf231-2e1c-4315-b3d5-ccee685c63b1 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:49:01.307899 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816bf231-2e1c-4315-b3d5-ccee685c63b1 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:49:01.308057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816bf231-2e1c-4315-b3d5-ccee685c63b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:49:01.326047 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-816bf231-2e1c-4315-b3d5-ccee685c63b1 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:49:01.327323 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-816bf231-2e1c-4315-b3d5-ccee685c63b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:49:01.391446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 604/1194] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:49:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:49:01.666624 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6e577d-da57-4a6a-8d3e-b1129653c17e None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:49:01.766257 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6e577d-da57-4a6a-8d3e-b1129653c17e None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:49:01.767734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:49:01.799023 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c293ab18-427f-46f4-8c22-0435e52307f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:49:01.799631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c293ab18-427f-46f4-8c22-0435e52307f6 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:49:01.799953 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c293ab18-427f-46f4-8c22-0435e52307f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:49:01.800253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c293ab18-427f-46f4-8c22-0435e52307f6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:49:01.809872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c293ab18-427f-46f4-8c22-0435e52307f6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:49:01.815104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-c293ab18-427f-46f4-8c22-0435e52307f6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:49:01.815104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:49:01.815104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:49:01.815104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:49:01.815104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:49:01.815104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:49:01.815104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:49:01.815104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:49:01.815104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:49:01.815104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:49:01.815104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:49:01.815104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:49:01.815104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:49:01.815104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:49:01.815104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:49:01.815104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:49:01.816184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:49:01.816184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:49:01.816184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:49:01.816184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:49:01.816184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:49:01.816184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:49:01.816184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:49:01.816184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:49:01.816184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:49:01.816184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:49:01.816184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:49:01.816184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:49:01.816184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:49:01.816184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:49:01.816184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:49:01.816184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:49:01.817213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:49:01.817213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:49:01.817213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:49:01.817213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:49:01.817213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:49:01.817213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:49:01.817213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:49:01.817213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:49:01.817213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:49:01.817213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:49:01.817213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:49:01.817213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:49:01.821229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 591/1195] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:49:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:49:01.822501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816bf231-2e1c-4315-b3d5-ccee685c63b1 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:49:01.867870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:49:01.921169 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816bf231-2e1c-4315-b3d5-ccee685c63b1 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:49:01.954828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca6300e-a784-4c80-bc30-24395376ed83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:49:01.955422 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca6300e-a784-4c80-bc30-24395376ed83 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:49:01.955702 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca6300e-a784-4c80-bc30-24395376ed83 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:49:01.955977 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca6300e-a784-4c80-bc30-24395376ed83 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:49:01.956637 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca6300e-a784-4c80-bc30-24395376ed83 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:49:01.956941 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca6300e-a784-4c80-bc30-24395376ed83 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:49:01.958348 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca6300e-a784-4c80-bc30-24395376ed83 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:49:01.966102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 605/1196] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:49:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:49:01.972144 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c293ab18-427f-46f4-8c22-0435e52307f6 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:49:02.069844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c293ab18-427f-46f4-8c22-0435e52307f6 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:49:02.102848 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e65ad-7bd6-4066-adb1-171e08ab6a93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:49:02.103458 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e65ad-7bd6-4066-adb1-171e08ab6a93 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:49:02.103738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e65ad-7bd6-4066-adb1-171e08ab6a93 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:49:02.104011 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e65ad-7bd6-4066-adb1-171e08ab6a93 None admin] PATH_INFO: `/v3/users/2def349673b14842a3ee9d67226cc3e8` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:49:02.105149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9e65ad-7bd6-4066-adb1-171e08ab6a93 None admin] RBAC: Authorizing `identity:get_user(user_id=2def349673b14842a3ee9d67226cc3e8)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:49:02.106559 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9e65ad-7bd6-4066-adb1-171e08ab6a93 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:49:02.108107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 592/1197] 10.222.0.42 () {56 vars in 1177 bytes} [Sun Sep 4 00:49:01 2022] GET /identity/v3/users/2def349673b14842a3ee9d67226cc3e8 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:49:02.114057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca6300e-a784-4c80-bc30-24395376ed83 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:49:02.212851 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca6300e-a784-4c80-bc30-24395376ed83 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:49:02.219660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:49:02.246147 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787fb1d1-e9ce-4a01-ab30-968d53f048b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:49:02.246939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787fb1d1-e9ce-4a01-ab30-968d53f048b6 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:49:02.247423 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787fb1d1-e9ce-4a01-ab30-968d53f048b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:49:02.247721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787fb1d1-e9ce-4a01-ab30-968d53f048b6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:49:02.316266 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787fb1d1-e9ce-4a01-ab30-968d53f048b6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:49:02.317749 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787fb1d1-e9ce-4a01-ab30-968d53f048b6 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:49:02.412530 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 606/1198] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:49:02 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 04 00:49:02.418641 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e65ad-7bd6-4066-adb1-171e08ab6a93 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:49:02.449815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:49:02.518436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e65ad-7bd6-4066-adb1-171e08ab6a93 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:49:02.551313 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492a2df5-7b66-44c5-b762-b8727321e4e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:49:02.551934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492a2df5-7b66-44c5-b762-b8727321e4e8 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:49:02.552212 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492a2df5-7b66-44c5-b762-b8727321e4e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:49:02.552484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492a2df5-7b66-44c5-b762-b8727321e4e8 None admin] PATH_INFO: `/v3/projects/166bb2b33981456dba5551ecb4010e91` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:49:02.553619 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492a2df5-7b66-44c5-b762-b8727321e4e8 None admin] RBAC: Authorizing `identity:get_project(project_id=166bb2b33981456dba5551ecb4010e91)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:49:02.555033 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492a2df5-7b66-44c5-b762-b8727321e4e8 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:49:02.556647 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 593/1199] 10.222.0.42 () {56 vars in 1186 bytes} [Sun Sep 4 00:49:02 2022] GET /identity/v3/projects/166bb2b33981456dba5551ecb4010e91 => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:49:02.563579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787fb1d1-e9ce-4a01-ab30-968d53f048b6 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:49:02.571589 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:49:02.672131 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787fb1d1-e9ce-4a01-ab30-968d53f048b6 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:49:02.705318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ccc97e-ee04-4abf-8537-9f4c101686f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:49:02.705919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ccc97e-ee04-4abf-8537-9f4c101686f5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:49:02.706199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ccc97e-ee04-4abf-8537-9f4c101686f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:49:02.706474 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ccc97e-ee04-4abf-8537-9f4c101686f5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:49:02.707197 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ccc97e-ee04-4abf-8537-9f4c101686f5 None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=166bb2b33981456dba5551ecb4010e91, user.id=2def349673b14842a3ee9d67226cc3e8) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:49:02.707504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ccc97e-ee04-4abf-8537-9f4c101686f5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:49:02.708923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ccc97e-ee04-4abf-8537-9f4c101686f5 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:49:02.714927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 607/1200] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:49:02 2022] GET /identity/v3/role_assignments?user.id=2def349673b14842a3ee9d67226cc3e8&scope.project.id=166bb2b33981456dba5551ecb4010e91&role.id=19c006412bec45dc8febd7cc86b9678f => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:49:12.547216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=51 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:49:12.712077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=65 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:13.437743 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ab77d7-8487-4716-b27f-bf5d48e5d8b5 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:13.438352 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ab77d7-8487-4716-b27f-bf5d48e5d8b5 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:13.438352 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ab77d7-8487-4716-b27f-bf5d48e5d8b5 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:13.439643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 594/1201] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:50:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:50:13.444830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb99074a-34bb-4d8c-86ce-e2f9c375e293 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:13.445249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb99074a-34bb-4d8c-86ce-e2f9c375e293 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:13.445413 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb99074a-34bb-4d8c-86ce-e2f9c375e293 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:13.463581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-bb99074a-34bb-4d8c-86ce-e2f9c375e293 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:50:13.464758 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-bb99074a-34bb-4d8c-86ce-e2f9c375e293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:50:13.531651 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 608/1202] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:50:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:50:13.800492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ab77d7-8487-4716-b27f-bf5d48e5d8b5 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:13.899578 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ab77d7-8487-4716-b27f-bf5d48e5d8b5 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:13.934106 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bdc822-c127-4217-a0ab-1efaf75022bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:13.934722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bdc822-c127-4217-a0ab-1efaf75022bd None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:13.934999 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bdc822-c127-4217-a0ab-1efaf75022bd None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:13.935267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bdc822-c127-4217-a0ab-1efaf75022bd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:13.945447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bdc822-c127-4217-a0ab-1efaf75022bd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:13.946935 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bdc822-c127-4217-a0ab-1efaf75022bd None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:13.961318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-06bdc822-c127-4217-a0ab-1efaf75022bd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:13.961318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:13.961318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:13.961318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:13.961318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:13.961318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:13.961318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:13.961318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:13.961318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:13.961318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:13.961318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:13.961318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:13.961318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:50:13.961318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:50:13.961318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:50:13.961318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:50:13.962519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:13.962519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:13.962519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:13.962519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:13.962519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:13.962519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:13.962519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:13.962519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:13.962519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:13.962519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:13.962519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:13.962519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:13.962519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:13.962519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:13.962519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:13.962519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:13.962519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:50:13.963647 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:50:13.963647 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:13.963647 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:50:13.963647 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 595/1203] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:13.969212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb99074a-34bb-4d8c-86ce-e2f9c375e293 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:14.066521 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb99074a-34bb-4d8c-86ce-e2f9c375e293 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:14.095009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:14.100781 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf58121-ce5b-4583-967d-546885f948fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:14.101443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf58121-ce5b-4583-967d-546885f948fe None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:14.101723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf58121-ce5b-4583-967d-546885f948fe None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:14.101997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf58121-ce5b-4583-967d-546885f948fe None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:14.102676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf58121-ce5b-4583-967d-546885f948fe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:14.102980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf58121-ce5b-4583-967d-546885f948fe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:14.104341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf58121-ce5b-4583-967d-546885f948fe None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:14.115832 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 609/1204] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:50:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:50:14.121652 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bdc822-c127-4217-a0ab-1efaf75022bd None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:14.174792 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:14.220770 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bdc822-c127-4217-a0ab-1efaf75022bd None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:14.253668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52c57ba-4c87-45bb-98c1-42042073b88e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:14.254263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52c57ba-4c87-45bb-98c1-42042073b88e None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:14.254541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52c57ba-4c87-45bb-98c1-42042073b88e None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:14.254812 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52c57ba-4c87-45bb-98c1-42042073b88e None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:14.255514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52c57ba-4c87-45bb-98c1-42042073b88e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:14.256921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52c57ba-4c87-45bb-98c1-42042073b88e None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:14.328592 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 596/1205] 10.222.0.42 () {60 vars in 1132 bytes} [Sun Sep 4 00:50:14 2022] POST /identity/v3/users => generated 251 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 00:50:16.260288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95904cec-8ec6-4fd8-8324-3528317158db None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:16.260841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95904cec-8ec6-4fd8-8324-3528317158db None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:16.260938 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95904cec-8ec6-4fd8-8324-3528317158db None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:16.262235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 610/1206] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:50:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:50:16.268148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a42cb2c-deb7-4646-8e80-daf9772872c0 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:16.268569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a42cb2c-deb7-4646-8e80-daf9772872c0 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:16.268687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a42cb2c-deb7-4646-8e80-daf9772872c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:16.286875 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-9a42cb2c-deb7-4646-8e80-daf9772872c0 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:50:16.288049 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-9a42cb2c-deb7-4646-8e80-daf9772872c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:50:16.353020 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 597/1207] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:50:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:50:16.621791 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95904cec-8ec6-4fd8-8324-3528317158db None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:16.728266 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95904cec-8ec6-4fd8-8324-3528317158db None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:16.762146 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac44ecd-4e02-4305-b256-2920dbabea3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:16.762760 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac44ecd-4e02-4305-b256-2920dbabea3d None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:16.763036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac44ecd-4e02-4305-b256-2920dbabea3d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:16.763304 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac44ecd-4e02-4305-b256-2920dbabea3d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:16.773051 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac44ecd-4e02-4305-b256-2920dbabea3d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:50:16.777825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-2ac44ecd-4e02-4305-b256-2920dbabea3d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:50:16.777825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:16.777825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:16.777825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:16.777825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:16.777825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:16.777825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:16.777825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:16.777825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:16.777825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:16.777825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:16.777825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:16.777825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:50:16.777825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:50:16.777825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:50:16.777825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:50:16.778956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:50:16.778956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:50:16.778956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:50:16.778956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:50:16.778956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:16.778956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:16.778956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:16.778956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:16.778956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:16.778956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:16.778956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:16.778956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:16.778956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:16.778956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:16.778956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:16.778956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:16.780698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:16.780698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:16.780698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:16.780698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:16.780698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:50:16.780698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:50:16.780698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:50:16.780698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:50:16.780698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:50:16.780698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:50:16.780698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:50:16.780698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:16.780698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 611/1208] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:50:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:16.785533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a42cb2c-deb7-4646-8e80-daf9772872c0 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:16.824195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:16.882446 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a42cb2c-deb7-4646-8e80-daf9772872c0 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:16.914717 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f80ebf3-033f-4af5-bc01-5cf4cafd86f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:16.915306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f80ebf3-033f-4af5-bc01-5cf4cafd86f9 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:16.915583 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f80ebf3-033f-4af5-bc01-5cf4cafd86f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:16.915851 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f80ebf3-033f-4af5-bc01-5cf4cafd86f9 None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:16.916515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f80ebf3-033f-4af5-bc01-5cf4cafd86f9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:16.916816 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f80ebf3-033f-4af5-bc01-5cf4cafd86f9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:16.918194 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f80ebf3-033f-4af5-bc01-5cf4cafd86f9 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:16.927161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 598/1209] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:50:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:16.932991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac44ecd-4e02-4305-b256-2920dbabea3d None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:16.981927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:17.028869 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac44ecd-4e02-4305-b256-2920dbabea3d None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:17.062088 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e7c301-d3d6-4e94-97ac-950267c52eb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:17.062699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e7c301-d3d6-4e94-97ac-950267c52eb7 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:17.062974 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e7c301-d3d6-4e94-97ac-950267c52eb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:17.063242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e7c301-d3d6-4e94-97ac-950267c52eb7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:17.072675 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e7c301-d3d6-4e94-97ac-950267c52eb7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:17.074124 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e7c301-d3d6-4e94-97ac-950267c52eb7 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:17.087965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-97e7c301-d3d6-4e94-97ac-950267c52eb7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:17.087965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:17.087965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:17.087965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:17.087965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:17.087965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:17.087965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:17.087965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:17.087965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:17.087965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:17.087965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:17.087965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:17.087965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:50:17.087965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:50:17.087965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:50:17.087965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:50:17.093174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:17.093174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:17.093174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:17.093174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:17.093174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:17.093174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:17.093174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:17.093174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:17.093174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:17.093174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:17.093174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:17.093174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:17.093174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:17.093174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:17.093174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:17.093174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:17.093174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:50:17.094285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:50:17.094285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:17.094285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:17.094285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 612/1210] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:17.095670 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f80ebf3-033f-4af5-bc01-5cf4cafd86f9 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:17.192328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:17.195060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f80ebf3-033f-4af5-bc01-5cf4cafd86f9 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:17.227966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343ee352-05e6-4370-99e0-c8b4ea8897a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:17.228565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343ee352-05e6-4370-99e0-c8b4ea8897a2 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:17.228843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343ee352-05e6-4370-99e0-c8b4ea8897a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:17.229139 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343ee352-05e6-4370-99e0-c8b4ea8897a2 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:17.229781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343ee352-05e6-4370-99e0-c8b4ea8897a2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:17.230086 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343ee352-05e6-4370-99e0-c8b4ea8897a2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:17.231456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343ee352-05e6-4370-99e0-c8b4ea8897a2 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:17.244481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 599/1211] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:50:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:17.250538 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e7c301-d3d6-4e94-97ac-950267c52eb7 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:17.356286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e7c301-d3d6-4e94-97ac-950267c52eb7 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:17.357928 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:17.390911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5797c328-5142-4a69-85ec-35d25f44c582 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:17.391532 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5797c328-5142-4a69-85ec-35d25f44c582 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:17.391807 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5797c328-5142-4a69-85ec-35d25f44c582 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:17.392072 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5797c328-5142-4a69-85ec-35d25f44c582 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:17.398748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5797c328-5142-4a69-85ec-35d25f44c582 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:17.400168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5797c328-5142-4a69-85ec-35d25f44c582 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:17.411943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-5797c328-5142-4a69-85ec-35d25f44c582 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 04 00:50:17.411943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:17.411943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:17.411943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:17.411943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:17.411943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:17.411943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:17.411943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:17.411943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:17.411943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:17.411943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:17.411943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:17.411943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:50:17.411943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:50:17.411943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:50:17.411943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:17.413102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:17.424499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:17.424499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:17.424499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:50:17.424499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:50:17.424499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:50:17.424499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:50:17.424499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:50:17.424499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:50:17.424499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 04 00:50:17.424499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:17.424499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 613/1212] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:17 2022] GET /identity/v3/users/placement => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:17.425275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343ee352-05e6-4370-99e0-c8b4ea8897a2 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:17.519520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343ee352-05e6-4370-99e0-c8b4ea8897a2 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:17.551728 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bfa91c-1df3-4d0d-9950-fdca14aabe86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:17.552339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bfa91c-1df3-4d0d-9950-fdca14aabe86 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:17.552628 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bfa91c-1df3-4d0d-9950-fdca14aabe86 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:17.552894 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bfa91c-1df3-4d0d-9950-fdca14aabe86 None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:17.553666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bfa91c-1df3-4d0d-9950-fdca14aabe86 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:17.558428 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bfa91c-1df3-4d0d-9950-fdca14aabe86 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:17.558428 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bfa91c-1df3-4d0d-9950-fdca14aabe86 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:17.572314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 600/1213] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Sep 4 00:50:17 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:17.578814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5797c328-5142-4a69-85ec-35d25f44c582 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:17.679162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5797c328-5142-4a69-85ec-35d25f44c582 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:17.712773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be789b00-ade2-4421-844a-b932ff6b555e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:17.713352 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be789b00-ade2-4421-844a-b932ff6b555e None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:17.713632 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be789b00-ade2-4421-844a-b932ff6b555e None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:17.713905 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be789b00-ade2-4421-844a-b932ff6b555e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:17.742517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:17.780134 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be789b00-ade2-4421-844a-b932ff6b555e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:17.781635 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be789b00-ade2-4421-844a-b932ff6b555e None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:17.879271 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 614/1214] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:50:17 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 04 00:50:17.885169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bfa91c-1df3-4d0d-9950-fdca14aabe86 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:17.904005 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:17.986510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bfa91c-1df3-4d0d-9950-fdca14aabe86 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:18.021607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f3a4ed-f07e-45ed-89c2-2aeacde2106c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:18.022228 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f3a4ed-f07e-45ed-89c2-2aeacde2106c None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:18.022504 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f3a4ed-f07e-45ed-89c2-2aeacde2106c None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:18.022776 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f3a4ed-f07e-45ed-89c2-2aeacde2106c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:18.032936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f3a4ed-f07e-45ed-89c2-2aeacde2106c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:18.034390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f3a4ed-f07e-45ed-89c2-2aeacde2106c None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:18.049278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-55f3a4ed-f07e-45ed-89c2-2aeacde2106c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:18.049278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:18.049278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:18.049278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:18.049278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:18.049278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:18.049278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:18.049278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:18.049278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:18.049278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:18.049278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:18.049278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:18.049278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:50:18.049278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:50:18.049278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:50:18.049278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:50:18.050441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:18.050441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:18.050441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:18.050441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:18.050441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:18.050441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:18.050441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:18.050441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:18.050441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:18.050441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:18.050441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:18.050441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:18.050441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:18.050441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:18.050441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:18.050441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:18.050441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:50:18.051526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:50:18.051526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:18.051526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:50:18.051526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 601/1215] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:18.056119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be789b00-ade2-4421-844a-b932ff6b555e None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:18.116305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:18.153908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be789b00-ade2-4421-844a-b932ff6b555e None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:18.187644 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f79aac8-077e-4921-ae0f-682353685173 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:18.188207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f79aac8-077e-4921-ae0f-682353685173 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:18.188487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f79aac8-077e-4921-ae0f-682353685173 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:18.188758 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f79aac8-077e-4921-ae0f-682353685173 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:18.189443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f79aac8-077e-4921-ae0f-682353685173 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:18.189739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f79aac8-077e-4921-ae0f-682353685173 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:18.191098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f79aac8-077e-4921-ae0f-682353685173 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:18.203013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 615/1216] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:50:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:18.208805 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f3a4ed-f07e-45ed-89c2-2aeacde2106c None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:18.279127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:18.309464 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f3a4ed-f07e-45ed-89c2-2aeacde2106c None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:18.341446 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7616d25e-f59a-4627-8586-48befdb1e203 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:18.342067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7616d25e-f59a-4627-8586-48befdb1e203 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:18.343039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7616d25e-f59a-4627-8586-48befdb1e203 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:18.343318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7616d25e-f59a-4627-8586-48befdb1e203 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:18.350004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7616d25e-f59a-4627-8586-48befdb1e203 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:18.355559 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7616d25e-f59a-4627-8586-48befdb1e203 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:18.363783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-7616d25e-f59a-4627-8586-48befdb1e203 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:50:18.363783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:18.363783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:18.363783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:18.363783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:18.363783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:18.363783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:18.363783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:18.363783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:18.363783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:18.363783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:18.363783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:18.363783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:50:18.363783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:50:18.363783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:50:18.364867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:50:18.364867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:18.364867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:18.364867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:18.364867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:18.364867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:18.364867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:18.364867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:18.364867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:18.364867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:18.364867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:18.364867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:18.364867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:18.364867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:18.364867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:18.364867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:18.364867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:18.366205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:50:18.366205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:50:18.366205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:50:18.366205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:50:18.366205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:50:18.366205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:50:18.366205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:50:18.366205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:50:18.366205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 602/1217] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:50:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:18.370830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f79aac8-077e-4921-ae0f-682353685173 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:18.470929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f79aac8-077e-4921-ae0f-682353685173 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:18.493096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:18.505211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da07123b-d957-4965-af3d-986b666d2630 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:18.505808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da07123b-d957-4965-af3d-986b666d2630 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:18.506085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da07123b-d957-4965-af3d-986b666d2630 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:18.506354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da07123b-d957-4965-af3d-986b666d2630 None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:18.507081 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da07123b-d957-4965-af3d-986b666d2630 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:18.507386 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da07123b-d957-4965-af3d-986b666d2630 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:18.508802 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da07123b-d957-4965-af3d-986b666d2630 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:18.520578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 616/1218] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:50:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:18.526844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7616d25e-f59a-4627-8586-48befdb1e203 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:18.626016 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7616d25e-f59a-4627-8586-48befdb1e203 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:18.653961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:18.660580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f0029c-42e8-4d9e-bfa6-3982f6003475 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:18.661205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f0029c-42e8-4d9e-bfa6-3982f6003475 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:18.661482 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f0029c-42e8-4d9e-bfa6-3982f6003475 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:18.661748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f0029c-42e8-4d9e-bfa6-3982f6003475 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:18.662472 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f0029c-42e8-4d9e-bfa6-3982f6003475 None admin] RBAC: Adding query filter params (role.id=df912a77fbaa479db172f76a214f8bc8, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=b29f444d56ec49308f7fef16f4d1eea5) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:18.662776 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f0029c-42e8-4d9e-bfa6-3982f6003475 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:18.664174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f0029c-42e8-4d9e-bfa6-3982f6003475 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:18.669498 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 603/1219] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:50:18 2022] GET /identity/v3/role_assignments?user.id=b29f444d56ec49308f7fef16f4d1eea5&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=df912a77fbaa479db172f76a214f8bc8 => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:20.597250 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05644b68-3bde-4e41-a0e1-5cc6481108d5 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:20.597768 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05644b68-3bde-4e41-a0e1-5cc6481108d5 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:20.597893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05644b68-3bde-4e41-a0e1-5cc6481108d5 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:20.599177 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 617/1220] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:50:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:50:20.604475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b55672-add6-41ba-ac81-e8a45613288b None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:20.604857 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b55672-add6-41ba-ac81-e8a45613288b None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:20.605022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b55672-add6-41ba-ac81-e8a45613288b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:20.623216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-f6b55672-add6-41ba-ac81-e8a45613288b None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:50:20.625268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f6b55672-add6-41ba-ac81-e8a45613288b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:50:20.692245 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 604/1221] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:50:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:50:20.961237 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05644b68-3bde-4e41-a0e1-5cc6481108d5 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:21.063559 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05644b68-3bde-4e41-a0e1-5cc6481108d5 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:21.097305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fad631c-0b5a-4177-baef-2483d93ffb55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:21.097925 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fad631c-0b5a-4177-baef-2483d93ffb55 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:21.098202 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fad631c-0b5a-4177-baef-2483d93ffb55 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:21.098473 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fad631c-0b5a-4177-baef-2483d93ffb55 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:21.108977 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fad631c-0b5a-4177-baef-2483d93ffb55 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:50:21.113836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-4fad631c-0b5a-4177-baef-2483d93ffb55 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:50:21.113836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:21.113836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:21.113836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:21.113836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:21.113836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:21.113836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:21.113836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:21.113836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:21.113836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:21.113836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:21.113836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:21.113836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:50:21.113836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:50:21.113836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:50:21.113836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:50:21.114799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:50:21.114799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:50:21.114799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:50:21.114799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:50:21.114799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:21.114799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:21.114799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:21.114799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:21.114799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:21.114799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:21.114799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:21.114799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:21.114799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:21.114799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:21.114799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:21.114799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:21.116093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:21.116093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:21.116093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:21.116093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:21.116093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:50:21.116093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:50:21.116093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:50:21.116093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:50:21.116093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:50:21.116093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:50:21.116093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:50:21.116093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:21.116093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 618/1222] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:50:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:21.120937 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b55672-add6-41ba-ac81-e8a45613288b None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:21.218859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b55672-add6-41ba-ac81-e8a45613288b None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:21.243307 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:21.256411 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b18fa37-6125-4bf6-9e04-db55aa4469fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:21.257018 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b18fa37-6125-4bf6-9e04-db55aa4469fc None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:21.257328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b18fa37-6125-4bf6-9e04-db55aa4469fc None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:21.257600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b18fa37-6125-4bf6-9e04-db55aa4469fc None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:21.258266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b18fa37-6125-4bf6-9e04-db55aa4469fc None admin] RBAC: Adding query filter params (name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:21.258580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b18fa37-6125-4bf6-9e04-db55aa4469fc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:21.260000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b18fa37-6125-4bf6-9e04-db55aa4469fc None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:21.268191 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 605/1223] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:50:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:21.274026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fad631c-0b5a-4177-baef-2483d93ffb55 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:21.318224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:21.374631 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fad631c-0b5a-4177-baef-2483d93ffb55 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:21.408734 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5b7f6b-46cb-491e-aae3-cfd7368959bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:21.409419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5b7f6b-46cb-491e-aae3-cfd7368959bb None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:21.409696 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5b7f6b-46cb-491e-aae3-cfd7368959bb None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:21.409966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5b7f6b-46cb-491e-aae3-cfd7368959bb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:21.420153 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5b7f6b-46cb-491e-aae3-cfd7368959bb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:21.422225 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5b7f6b-46cb-491e-aae3-cfd7368959bb None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:21.437736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-6c5b7f6b-46cb-491e-aae3-cfd7368959bb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:21.437736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:21.437736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:21.437736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:21.437736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:21.437736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:21.437736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:21.437736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:21.437736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:21.437736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:21.437736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:21.437736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:21.437736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:50:21.437736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:50:21.437736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:50:21.437736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:50:21.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:21.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:21.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:21.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:21.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:21.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:21.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:21.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:21.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:21.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:21.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:21.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:21.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:21.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:21.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:21.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:21.438932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:50:21.439843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:50:21.439843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:21.439843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:21.439843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 619/1224] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:21.444591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b18fa37-6125-4bf6-9e04-db55aa4469fc None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:21.543006 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b18fa37-6125-4bf6-9e04-db55aa4469fc None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:21.575351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cd4b65-c2a2-4566-b072-f1be8f7539c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:21.575945 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cd4b65-c2a2-4566-b072-f1be8f7539c5 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:21.576222 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cd4b65-c2a2-4566-b072-f1be8f7539c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:21.576515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cd4b65-c2a2-4566-b072-f1be8f7539c5 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:21.577181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cd4b65-c2a2-4566-b072-f1be8f7539c5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:21.577485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cd4b65-c2a2-4566-b072-f1be8f7539c5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:21.578830 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cd4b65-c2a2-4566-b072-f1be8f7539c5 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:21.590022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 606/1225] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:50:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:21.596357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5b7f6b-46cb-491e-aae3-cfd7368959bb None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:21.690431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:21.696953 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5b7f6b-46cb-491e-aae3-cfd7368959bb None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:21.730512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e0990c-a59d-4cb6-8ca7-e2228cc71297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:21.731139 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e0990c-a59d-4cb6-8ca7-e2228cc71297 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:21.731421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e0990c-a59d-4cb6-8ca7-e2228cc71297 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:21.731697 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e0990c-a59d-4cb6-8ca7-e2228cc71297 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:21.738992 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e0990c-a59d-4cb6-8ca7-e2228cc71297 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:21.742327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e0990c-a59d-4cb6-8ca7-e2228cc71297 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:21.752264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-92e0990c-a59d-4cb6-8ca7-e2228cc71297 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 04 00:50:21.752264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:21.752264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:21.752264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:21.752264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:21.752264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:21.752264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:21.752264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:21.752264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:21.752264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:21.752264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:21.752264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:21.752264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:50:21.752264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:50:21.752264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:50:21.752264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:21.753489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:21.754908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:21.754908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:21.754908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:50:21.754908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:50:21.754908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:50:21.754908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:50:21.754908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:50:21.754908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:50:21.754908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 04 00:50:21.754908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:21.754908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 620/1226] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:21 2022] GET /identity/v3/users/placement => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:21.759605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cd4b65-c2a2-4566-b072-f1be8f7539c5 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:21.806829 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:21.859790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cd4b65-c2a2-4566-b072-f1be8f7539c5 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:21.893751 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a30930-6d5c-41d6-9f85-f5211632fd88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:21.894351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a30930-6d5c-41d6-9f85-f5211632fd88 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:21.894632 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a30930-6d5c-41d6-9f85-f5211632fd88 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:21.894915 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a30930-6d5c-41d6-9f85-f5211632fd88 None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:21.895638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a30930-6d5c-41d6-9f85-f5211632fd88 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:21.895943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a30930-6d5c-41d6-9f85-f5211632fd88 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:21.897386 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a30930-6d5c-41d6-9f85-f5211632fd88 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:21.914412 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 607/1227] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Sep 4 00:50:21 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:50:21.920140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e0990c-a59d-4cb6-8ca7-e2228cc71297 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:22.020284 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e0990c-a59d-4cb6-8ca7-e2228cc71297 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:22.052887 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cfe652-eac5-4c1b-a8b7-e7705e2b15cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:22.053539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cfe652-eac5-4c1b-a8b7-e7705e2b15cc None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:22.053819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cfe652-eac5-4c1b-a8b7-e7705e2b15cc None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:22.054091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cfe652-eac5-4c1b-a8b7-e7705e2b15cc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:22.064913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3cfe652-eac5-4c1b-a8b7-e7705e2b15cc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:22.066339 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3cfe652-eac5-4c1b-a8b7-e7705e2b15cc None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:22.081142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-e3cfe652-eac5-4c1b-a8b7-e7705e2b15cc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:22.081142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:22.081142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:22.081142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:22.081142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:22.081142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:22.081142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:22.081142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:22.081142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:22.081142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:22.081142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:22.081142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:22.081142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:50:22.081142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:50:22.081142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:50:22.081142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:50:22.082329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:22.082329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:22.082329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:22.082329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:22.082329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:22.082329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:22.082329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:22.082329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:22.082329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:22.082329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:22.082329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:22.082329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:22.082329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:22.082329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:22.082329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:22.082329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:22.082329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:50:22.083500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:50:22.083500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:22.083500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:22.083500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 621/1228] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:22.088810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a30930-6d5c-41d6-9f85-f5211632fd88 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:22.193496 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a30930-6d5c-41d6-9f85-f5211632fd88 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:22.195789 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:22.226053 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc00b4d7-e5ee-4570-ae4a-fbd7affd9d04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:22.226646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc00b4d7-e5ee-4570-ae4a-fbd7affd9d04 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:22.226923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc00b4d7-e5ee-4570-ae4a-fbd7affd9d04 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:22.227191 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc00b4d7-e5ee-4570-ae4a-fbd7affd9d04 None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:22.227828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc00b4d7-e5ee-4570-ae4a-fbd7affd9d04 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:22.228125 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc00b4d7-e5ee-4570-ae4a-fbd7affd9d04 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:22.229513 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc00b4d7-e5ee-4570-ae4a-fbd7affd9d04 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:22.240135 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 608/1229] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:50:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:22.245924 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cfe652-eac5-4c1b-a8b7-e7705e2b15cc None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:22.256174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:22.344245 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cfe652-eac5-4c1b-a8b7-e7705e2b15cc None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:22.378401 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086ab45a-5977-4d47-8394-ce9f6b45b69b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:22.379053 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086ab45a-5977-4d47-8394-ce9f6b45b69b None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:22.379431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086ab45a-5977-4d47-8394-ce9f6b45b69b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:22.379634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086ab45a-5977-4d47-8394-ce9f6b45b69b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:22.386219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086ab45a-5977-4d47-8394-ce9f6b45b69b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:22.387694 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086ab45a-5977-4d47-8394-ce9f6b45b69b None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:22.399831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-086ab45a-5977-4d47-8394-ce9f6b45b69b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:50:22.399831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:22.399831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:22.399831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:22.399831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:22.399831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:22.399831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:22.399831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:22.399831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:22.399831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:22.399831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:22.399831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:22.399831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:50:22.399831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:50:22.399831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:50:22.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:50:22.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:22.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:22.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:22.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:22.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:22.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:22.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:22.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:22.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:22.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:22.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:22.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:22.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:22.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:22.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:22.405579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:22.407930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:50:22.407930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:50:22.407930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:50:22.407930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:50:22.407930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:50:22.407930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:50:22.407930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:50:22.407930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:22.407930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 622/1230] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:50:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:22.408549 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc00b4d7-e5ee-4570-ae4a-fbd7affd9d04 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:22.510052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc00b4d7-e5ee-4570-ae4a-fbd7affd9d04 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:22.542996 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2e1fdd-9cd8-4c64-8da3-e721e1e03133 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:22.543593 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2e1fdd-9cd8-4c64-8da3-e721e1e03133 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:22.543908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2e1fdd-9cd8-4c64-8da3-e721e1e03133 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:22.544223 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2e1fdd-9cd8-4c64-8da3-e721e1e03133 None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:22.544898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2e1fdd-9cd8-4c64-8da3-e721e1e03133 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:22.545762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2e1fdd-9cd8-4c64-8da3-e721e1e03133 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:22.547362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2e1fdd-9cd8-4c64-8da3-e721e1e03133 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:22.559715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 609/1231] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:50:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:50:22.566602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086ab45a-5977-4d47-8394-ce9f6b45b69b None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:22.632673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:22.666020 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086ab45a-5977-4d47-8394-ce9f6b45b69b None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:22.701082 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a36ac7-b4ea-4e24-a65b-c72855753ebe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:22.701968 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a36ac7-b4ea-4e24-a65b-c72855753ebe None admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:22.702179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a36ac7-b4ea-4e24-a65b-c72855753ebe None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:22.702447 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a36ac7-b4ea-4e24-a65b-c72855753ebe None admin] PATH_INFO: `/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/b29f444d56ec49308f7fef16f4d1eea5/roles/df912a77fbaa479db172f76a214f8bc8` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:22.716662 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a36ac7-b4ea-4e24-a65b-c72855753ebe None admin] RBAC: Authorizing `identity:create_grant(project_id=b1f56314d44d4faaa88e735ea196783f, user_id=b29f444d56ec49308f7fef16f4d1eea5, role_id=df912a77fbaa479db172f76a214f8bc8)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:22.718694 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a36ac7-b4ea-4e24-a65b-c72855753ebe None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:22.783520 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 623/1232] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:50:22 2022] PUT /identity/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/b29f444d56ec49308f7fef16f4d1eea5/roles/df912a77fbaa479db172f76a214f8bc8 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 00:50:24.692962 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b8f8c0-23df-4089-bff0-bd8351863949 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:24.693527 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b8f8c0-23df-4089-bff0-bd8351863949 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:24.693527 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b8f8c0-23df-4089-bff0-bd8351863949 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:24.694865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 610/1233] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:50:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:50:24.700129 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937e6a2a-bda6-4668-84ba-6ff0b0abb710 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:24.700546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937e6a2a-bda6-4668-84ba-6ff0b0abb710 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:24.700694 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937e6a2a-bda6-4668-84ba-6ff0b0abb710 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:24.721613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-937e6a2a-bda6-4668-84ba-6ff0b0abb710 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:50:24.722736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-937e6a2a-bda6-4668-84ba-6ff0b0abb710 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:50:24.788998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 624/1234] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:50:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:50:25.065354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b8f8c0-23df-4089-bff0-bd8351863949 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:25.091680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:25.173186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b8f8c0-23df-4089-bff0-bd8351863949 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:25.207124 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706af294-dded-42ac-bed3-5199d4740fdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:25.207744 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706af294-dded-42ac-bed3-5199d4740fdd None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:25.208022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706af294-dded-42ac-bed3-5199d4740fdd None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:25.208295 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706af294-dded-42ac-bed3-5199d4740fdd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:25.218239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706af294-dded-42ac-bed3-5199d4740fdd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:50:25.222936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-706af294-dded-42ac-bed3-5199d4740fdd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:50:25.222936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:25.222936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:25.222936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:25.222936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:25.222936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:25.222936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:25.222936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:25.222936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:25.222936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:25.222936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:25.222936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:25.222936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:50:25.222936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:50:25.222936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:50:25.222936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:50:25.223878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:50:25.223878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:50:25.223878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:50:25.223878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:50:25.223878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:25.223878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:25.223878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:25.223878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:25.223878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:25.223878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:25.223878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:25.223878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:25.223878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:25.223878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:25.223878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:25.223878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:25.224769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:25.224769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:25.224769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:25.224769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:25.224769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:50:25.224769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:50:25.224769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:50:25.224769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:50:25.224769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:50:25.224769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:50:25.224769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 04 00:50:25.224769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:50:25.224769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 611/1235] 10.222.0.42 () {56 vars in 1102 bytes} [Sun Sep 4 00:50:25 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:25.230122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937e6a2a-bda6-4668-84ba-6ff0b0abb710 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:25.277301 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:25.329468 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937e6a2a-bda6-4668-84ba-6ff0b0abb710 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:25.362330 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03936168-a0f7-49db-a3e8-ccadf659875f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:25.362922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03936168-a0f7-49db-a3e8-ccadf659875f None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:25.363196 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03936168-a0f7-49db-a3e8-ccadf659875f None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:25.363462 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03936168-a0f7-49db-a3e8-ccadf659875f None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:25.364113 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03936168-a0f7-49db-a3e8-ccadf659875f None admin] RBAC: Adding query filter params (name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:25.364410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03936168-a0f7-49db-a3e8-ccadf659875f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:25.365797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03936168-a0f7-49db-a3e8-ccadf659875f None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:25.374317 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 625/1236] 10.222.0.42 () {56 vars in 1103 bytes} [Sun Sep 4 00:50:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:50:25.380140 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706af294-dded-42ac-bed3-5199d4740fdd None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:25.467050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:25.482969 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706af294-dded-42ac-bed3-5199d4740fdd None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:25.516261 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7185d3dd-0735-4e18-a3f9-d66787f57cea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:25.516874 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7185d3dd-0735-4e18-a3f9-d66787f57cea None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:25.517181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7185d3dd-0735-4e18-a3f9-d66787f57cea None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:25.517451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7185d3dd-0735-4e18-a3f9-d66787f57cea None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:25.527171 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7185d3dd-0735-4e18-a3f9-d66787f57cea None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:25.528589 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7185d3dd-0735-4e18-a3f9-d66787f57cea None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:25.542673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-7185d3dd-0735-4e18-a3f9-d66787f57cea None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:25.542673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:25.542673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:25.542673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:25.542673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:25.542673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:25.542673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:25.542673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:25.542673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:25.542673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:25.542673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:25.542673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:25.542673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:50:25.542673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:50:25.542673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:50:25.542673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:50:25.543665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:25.543665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:25.543665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:25.543665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:25.543665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:25.543665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:25.543665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:25.543665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:25.543665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:25.543665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:25.543665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:25.543665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:25.543665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:25.543665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:25.543665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:25.543665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:25.543665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:50:25.544883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:50:25.544883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:25.544883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:50:25.544883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 612/1237] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:25.550491 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03936168-a0f7-49db-a3e8-ccadf659875f None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:25.650827 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03936168-a0f7-49db-a3e8-ccadf659875f None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:25.659903 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:25.684220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6785051f-aa2e-4b0c-bf60-e8500debbd0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:25.684811 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6785051f-aa2e-4b0c-bf60-e8500debbd0a None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:25.685107 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6785051f-aa2e-4b0c-bf60-e8500debbd0a None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:25.685384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6785051f-aa2e-4b0c-bf60-e8500debbd0a None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:25.686613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6785051f-aa2e-4b0c-bf60-e8500debbd0a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:25.686910 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6785051f-aa2e-4b0c-bf60-e8500debbd0a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:25.688259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6785051f-aa2e-4b0c-bf60-e8500debbd0a None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:25.699443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 626/1238] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:50:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:25.705457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7185d3dd-0735-4e18-a3f9-d66787f57cea None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:25.806193 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7185d3dd-0735-4e18-a3f9-d66787f57cea None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:25.838186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de32838-45b8-4664-b9ba-d23f32483e8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:25.838805 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de32838-45b8-4664-b9ba-d23f32483e8b None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:25.839086 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de32838-45b8-4664-b9ba-d23f32483e8b None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:25.839360 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de32838-45b8-4664-b9ba-d23f32483e8b None admin] PATH_INFO: `/v3/users/placement` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:25.842623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:25.847689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de32838-45b8-4664-b9ba-d23f32483e8b None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:25.849148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de32838-45b8-4664-b9ba-d23f32483e8b None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:25.861141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-1de32838-45b8-4664-b9ba-d23f32483e8b None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 04 00:50:25.861141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:25.861141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:25.861141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:25.861141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:25.861141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:25.861141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:25.861141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:25.861141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:25.861141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:25.861141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:25.861141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:25.861141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:50:25.861141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:50:25.861141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:50:25.861141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:25.862260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:25.863459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:25.863459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:25.863459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:50:25.863459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:50:25.863459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:50:25.863459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:50:25.863459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:50:25.863459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:50:25.863459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 04 00:50:25.863459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:50:25.865241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 613/1239] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:25 2022] GET /identity/v3/users/placement => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:25.868050 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6785051f-aa2e-4b0c-bf60-e8500debbd0a None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:25.965136 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6785051f-aa2e-4b0c-bf60-e8500debbd0a None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:26.000533 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64adede5-00ac-4bae-8dc6-bb9b4d597bc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:26.001146 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64adede5-00ac-4bae-8dc6-bb9b4d597bc7 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:26.001425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64adede5-00ac-4bae-8dc6-bb9b4d597bc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:26.001692 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64adede5-00ac-4bae-8dc6-bb9b4d597bc7 None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:26.002405 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64adede5-00ac-4bae-8dc6-bb9b4d597bc7 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:26.002706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64adede5-00ac-4bae-8dc6-bb9b4d597bc7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:26.004159 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64adede5-00ac-4bae-8dc6-bb9b4d597bc7 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:26.025669 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 627/1240] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Sep 4 00:50:25 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:26.032092 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de32838-45b8-4664-b9ba-d23f32483e8b None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:26.131437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de32838-45b8-4664-b9ba-d23f32483e8b None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:26.163973 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551d9e3a-dd97-4fc6-acad-cf6b3a27c992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:26.164566 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551d9e3a-dd97-4fc6-acad-cf6b3a27c992 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:26.164853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551d9e3a-dd97-4fc6-acad-cf6b3a27c992 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:26.165155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551d9e3a-dd97-4fc6-acad-cf6b3a27c992 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:26.225803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:26.232000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551d9e3a-dd97-4fc6-acad-cf6b3a27c992 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:26.234108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551d9e3a-dd97-4fc6-acad-cf6b3a27c992 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:26.328460 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 614/1241] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:50:26 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 04 00:50:26.334497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64adede5-00ac-4bae-8dc6-bb9b4d597bc7 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:26.362039 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:26.435361 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64adede5-00ac-4bae-8dc6-bb9b4d597bc7 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:26.470119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3478f329-d598-4c2a-aacc-5b04eedb3cb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:26.470750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3478f329-d598-4c2a-aacc-5b04eedb3cb7 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:26.471025 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3478f329-d598-4c2a-aacc-5b04eedb3cb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:26.471293 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3478f329-d598-4c2a-aacc-5b04eedb3cb7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:26.481057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3478f329-d598-4c2a-aacc-5b04eedb3cb7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:26.483089 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3478f329-d598-4c2a-aacc-5b04eedb3cb7 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:26.497211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-3478f329-d598-4c2a-aacc-5b04eedb3cb7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:26.497211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:26.497211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:26.497211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:26.497211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:26.497211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:26.497211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:26.497211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:26.497211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:26.497211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:26.497211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:26.497211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:26.497211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:50:26.497211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:50:26.497211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:50:26.497211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:50:26.499310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:26.499310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:26.499310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:26.499310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:26.499310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:26.499310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:26.499310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:26.499310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:26.499310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:26.499310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:26.499310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:26.499310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:26.499310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:26.499310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:26.499310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:26.499310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:26.499310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:50:26.500276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:50:26.500276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:26.500276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:26.500276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 628/1242] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:26.504533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551d9e3a-dd97-4fc6-acad-cf6b3a27c992 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:26.596628 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:26.602328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551d9e3a-dd97-4fc6-acad-cf6b3a27c992 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:26.635820 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00418c6-28cc-400a-9843-dbac6aa56c2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:26.636520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00418c6-28cc-400a-9843-dbac6aa56c2f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:26.636803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00418c6-28cc-400a-9843-dbac6aa56c2f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:26.637092 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00418c6-28cc-400a-9843-dbac6aa56c2f None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:26.637744 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00418c6-28cc-400a-9843-dbac6aa56c2f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:26.638049 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00418c6-28cc-400a-9843-dbac6aa56c2f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:26.639532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00418c6-28cc-400a-9843-dbac6aa56c2f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:26.651060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 615/1243] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:50:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:26.657214 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3478f329-d598-4c2a-aacc-5b04eedb3cb7 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:26.740190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:26.760878 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3478f329-d598-4c2a-aacc-5b04eedb3cb7 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:26.793551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09188e1b-56a2-4583-89b4-be5994d33dd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:26.794200 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09188e1b-56a2-4583-89b4-be5994d33dd1 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:26.794478 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09188e1b-56a2-4583-89b4-be5994d33dd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:26.794755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09188e1b-56a2-4583-89b4-be5994d33dd1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:26.802601 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09188e1b-56a2-4583-89b4-be5994d33dd1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:26.804026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09188e1b-56a2-4583-89b4-be5994d33dd1 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:26.815127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-09188e1b-56a2-4583-89b4-be5994d33dd1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:50:26.815127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:26.815127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:26.815127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:26.815127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:26.815127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:26.815127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:26.815127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:26.815127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:26.815127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:26.815127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:26.815127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:26.815127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:50:26.815127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:50:26.815127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:50:26.816203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:50:26.816203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:26.816203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:26.816203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:26.816203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:26.816203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:26.816203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:26.816203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:26.816203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:26.816203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:26.816203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:26.816203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:26.816203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:26.816203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:26.816203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:26.816203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:26.816203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:26.817186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:50:26.817186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:50:26.817186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:50:26.817186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:50:26.817186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:50:26.817186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:50:26.817186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:50:26.817186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:26.817186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 629/1244] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:50:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:26.822265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00418c6-28cc-400a-9843-dbac6aa56c2f None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:26.921120 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00418c6-28cc-400a-9843-dbac6aa56c2f None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:26.954307 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb2d4f1-50b4-4927-8799-b7e324b759f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:26.954912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb2d4f1-50b4-4927-8799-b7e324b759f5 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:26.955191 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb2d4f1-50b4-4927-8799-b7e324b759f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:26.955458 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb2d4f1-50b4-4927-8799-b7e324b759f5 None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:26.956144 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb2d4f1-50b4-4927-8799-b7e324b759f5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:26.956448 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb2d4f1-50b4-4927-8799-b7e324b759f5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:26.957880 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb2d4f1-50b4-4927-8799-b7e324b759f5 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:26.970155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 616/1245] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:50:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:26.976348 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09188e1b-56a2-4583-89b4-be5994d33dd1 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:27.080687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09188e1b-56a2-4583-89b4-be5994d33dd1 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:27.114849 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23665cbb-f1f4-4372-8950-1865cf20321e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:27.115445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23665cbb-f1f4-4372-8950-1865cf20321e None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:27.115721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23665cbb-f1f4-4372-8950-1865cf20321e None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:27.115989 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23665cbb-f1f4-4372-8950-1865cf20321e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:27.116757 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23665cbb-f1f4-4372-8950-1865cf20321e None admin] RBAC: Adding query filter params (role.id=df912a77fbaa479db172f76a214f8bc8, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=b29f444d56ec49308f7fef16f4d1eea5) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:27.117125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23665cbb-f1f4-4372-8950-1865cf20321e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:27.118544 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23665cbb-f1f4-4372-8950-1865cf20321e None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:27.124585 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 630/1246] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:50:26 2022] GET /identity/v3/role_assignments?user.id=b29f444d56ec49308f7fef16f4d1eea5&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=df912a77fbaa479db172f76a214f8bc8 => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:29.086320 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547e7b41-f262-430d-82e9-0b3891baf3bd None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:29.086853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547e7b41-f262-430d-82e9-0b3891baf3bd None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:29.086853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547e7b41-f262-430d-82e9-0b3891baf3bd None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:29.088208 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 617/1247] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:50:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:50:29.094090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5e5315-6900-4c61-ab84-0538c42a254d None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:29.094516 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5e5315-6900-4c61-ab84-0538c42a254d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:29.094701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5e5315-6900-4c61-ab84-0538c42a254d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:29.101146 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:29.114031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-5b5e5315-6900-4c61-ab84-0538c42a254d None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:50:29.115277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-5b5e5315-6900-4c61-ab84-0538c42a254d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:50:29.180748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 631/1248] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:50:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:50:29.457404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547e7b41-f262-430d-82e9-0b3891baf3bd None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:29.476102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:29.558222 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547e7b41-f262-430d-82e9-0b3891baf3bd None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:29.591368 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b0257f-63bb-4931-b110-26c13d5d8aac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:29.592489 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b0257f-63bb-4931-b110-26c13d5d8aac None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:29.592782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b0257f-63bb-4931-b110-26c13d5d8aac None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:29.593031 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b0257f-63bb-4931-b110-26c13d5d8aac None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:29.603110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b0257f-63bb-4931-b110-26c13d5d8aac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:50:29.608514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-32b0257f-63bb-4931-b110-26c13d5d8aac None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:50:29.608514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:29.608514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:29.608514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:29.608514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:29.608514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:29.608514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:29.608514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:29.608514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:29.608514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:29.608514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:29.608514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:29.608514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:50:29.608514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:50:29.608514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:50:29.608514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:50:29.613017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:50:29.613017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:50:29.613017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:50:29.613017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:50:29.613017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:29.613017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:29.613017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:29.613017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:29.613017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:29.613017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:29.613017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:29.613017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:29.613017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:29.613017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:29.613017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:29.613017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:29.614137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:29.614137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:29.614137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:29.614137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:29.614137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:50:29.614137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:50:29.614137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:50:29.614137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:50:29.614137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:50:29.614137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:50:29.614137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:50:29.614137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:50:29.614137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 618/1249] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:50:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:29.615788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5e5315-6900-4c61-ab84-0538c42a254d None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:29.714270 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5e5315-6900-4c61-ab84-0538c42a254d None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:29.741778 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:29.746449 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9385492-5b8c-4d84-b77a-de2a0a7fbfe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:29.747049 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9385492-5b8c-4d84-b77a-de2a0a7fbfe6 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:29.747323 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9385492-5b8c-4d84-b77a-de2a0a7fbfe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:29.747590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9385492-5b8c-4d84-b77a-de2a0a7fbfe6 None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:29.748767 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9385492-5b8c-4d84-b77a-de2a0a7fbfe6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:29.749067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9385492-5b8c-4d84-b77a-de2a0a7fbfe6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:29.750447 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9385492-5b8c-4d84-b77a-de2a0a7fbfe6 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:29.758132 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 632/1250] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:50:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:29.764703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b0257f-63bb-4931-b110-26c13d5d8aac None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:29.841878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:29.864029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b0257f-63bb-4931-b110-26c13d5d8aac None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:29.898636 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eea6dac-feb9-4119-a28a-ea2b19a68775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:29.899249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eea6dac-feb9-4119-a28a-ea2b19a68775 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:29.899523 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eea6dac-feb9-4119-a28a-ea2b19a68775 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:29.899791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eea6dac-feb9-4119-a28a-ea2b19a68775 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:29.909374 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eea6dac-feb9-4119-a28a-ea2b19a68775 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:29.910790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eea6dac-feb9-4119-a28a-ea2b19a68775 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:29.924957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-6eea6dac-feb9-4119-a28a-ea2b19a68775 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:29.924957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:29.924957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:29.924957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:29.924957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:29.924957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:29.924957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:29.924957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:29.924957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:29.924957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:29.924957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:29.924957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:29.924957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:50:29.924957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:50:29.924957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:50:29.924957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:50:29.929586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:29.929586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:29.929586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:29.929586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:29.929586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:29.929586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:29.929586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:29.929586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:29.929586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:29.929586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:29.929586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:29.929586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:29.929586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:29.929586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:29.929586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:29.929586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:29.929586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:50:29.930804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:50:29.930804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:29.930804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:50:29.930804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 619/1251] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:29.932308 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9385492-5b8c-4d84-b77a-de2a0a7fbfe6 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:30.031285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9385492-5b8c-4d84-b77a-de2a0a7fbfe6 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:30.064264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e309c94-f7e5-4b5e-8e0f-8fd9ad7ff123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:30.064848 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e309c94-f7e5-4b5e-8e0f-8fd9ad7ff123 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:30.065158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e309c94-f7e5-4b5e-8e0f-8fd9ad7ff123 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:30.065431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e309c94-f7e5-4b5e-8e0f-8fd9ad7ff123 None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:30.066063 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e309c94-f7e5-4b5e-8e0f-8fd9ad7ff123 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:30.066362 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e309c94-f7e5-4b5e-8e0f-8fd9ad7ff123 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:30.067720 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e309c94-f7e5-4b5e-8e0f-8fd9ad7ff123 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:30.078948 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 633/1252] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:50:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:30.084622 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eea6dac-feb9-4119-a28a-ea2b19a68775 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:30.181375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eea6dac-feb9-4119-a28a-ea2b19a68775 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:30.207141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:30.214707 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a4d585-d72c-44bb-8a68-3f0c3a7dc2a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:30.215325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a4d585-d72c-44bb-8a68-3f0c3a7dc2a0 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:30.215601 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a4d585-d72c-44bb-8a68-3f0c3a7dc2a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:30.215869 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a4d585-d72c-44bb-8a68-3f0c3a7dc2a0 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:30.223581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a4d585-d72c-44bb-8a68-3f0c3a7dc2a0 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:30.224997 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a4d585-d72c-44bb-8a68-3f0c3a7dc2a0 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:30.236284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-48a4d585-d72c-44bb-8a68-3f0c3a7dc2a0 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 04 00:50:30.236284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:30.236284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:30.236284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:30.236284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:30.236284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:30.236284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:30.236284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:30.236284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:30.236284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:30.236284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:30.236284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:30.236284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:50:30.236284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:50:30.236284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:50:30.236284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:30.237447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:30.238600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:30.238600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:30.238600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:50:30.238600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:50:30.238600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:50:30.238600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:50:30.238600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:50:30.238600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:50:30.238600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 04 00:50:30.238600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:50:30.238600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 620/1253] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:30 2022] GET /identity/v3/users/placement => generated 87 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:30.243159 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e309c94-f7e5-4b5e-8e0f-8fd9ad7ff123 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:30.299248 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:30.341489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e309c94-f7e5-4b5e-8e0f-8fd9ad7ff123 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:30.377498 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8832a880-fc86-441f-b2b9-86d61c1466ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:30.378101 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8832a880-fc86-441f-b2b9-86d61c1466ac None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:30.378404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8832a880-fc86-441f-b2b9-86d61c1466ac None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:30.378670 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8832a880-fc86-441f-b2b9-86d61c1466ac None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:30.379392 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8832a880-fc86-441f-b2b9-86d61c1466ac None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:30.379694 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8832a880-fc86-441f-b2b9-86d61c1466ac None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:30.381207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8832a880-fc86-441f-b2b9-86d61c1466ac None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:30.398346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 634/1254] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Sep 4 00:50:30 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:30.404837 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a4d585-d72c-44bb-8a68-3f0c3a7dc2a0 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:30.505545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a4d585-d72c-44bb-8a68-3f0c3a7dc2a0 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:30.538900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82983b4-427b-4c41-8b98-903b8a03dc83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:30.539503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82983b4-427b-4c41-8b98-903b8a03dc83 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:30.539783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82983b4-427b-4c41-8b98-903b8a03dc83 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:30.540054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82983b4-427b-4c41-8b98-903b8a03dc83 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:30.593907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:30.605648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82983b4-427b-4c41-8b98-903b8a03dc83 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:30.607075 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82983b4-427b-4c41-8b98-903b8a03dc83 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:30.705342 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 621/1255] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:50:30 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 04 00:50:30.711493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8832a880-fc86-441f-b2b9-86d61c1466ac None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:30.811007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8832a880-fc86-441f-b2b9-86d61c1466ac None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:30.823716 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:30.845193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c009a48e-8613-4b62-b755-f4206cb40b0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:30.845756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c009a48e-8613-4b62-b755-f4206cb40b0a None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:30.846034 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c009a48e-8613-4b62-b755-f4206cb40b0a None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:30.846310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c009a48e-8613-4b62-b755-f4206cb40b0a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:30.856048 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c009a48e-8613-4b62-b755-f4206cb40b0a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:30.860079 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c009a48e-8613-4b62-b755-f4206cb40b0a None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:30.871618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-c009a48e-8613-4b62-b755-f4206cb40b0a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:30.871618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:30.871618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:30.871618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:30.871618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:30.871618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:30.871618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:30.871618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:30.871618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:30.871618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:30.871618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:30.871618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:30.871618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:50:30.871618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:50:30.871618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:50:30.871618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:50:30.876232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:30.876232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:30.876232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:30.876232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:30.876232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:30.876232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:30.876232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:30.876232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:30.876232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:30.876232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:30.876232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:30.876232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:30.876232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:30.876232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:30.876232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:30.876232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:30.876232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:50:30.877563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:50:30.877563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:30.877563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:30.877563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 635/1256] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:30.879006 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82983b4-427b-4c41-8b98-903b8a03dc83 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:30.968673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:30.980306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82983b4-427b-4c41-8b98-903b8a03dc83 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:31.014643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c72f8a3-910c-47c6-8cbc-c09da7e6f4fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:31.015280 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c72f8a3-910c-47c6-8cbc-c09da7e6f4fe None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:31.015566 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c72f8a3-910c-47c6-8cbc-c09da7e6f4fe None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:31.015841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c72f8a3-910c-47c6-8cbc-c09da7e6f4fe None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:31.016481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c72f8a3-910c-47c6-8cbc-c09da7e6f4fe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:31.016786 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c72f8a3-910c-47c6-8cbc-c09da7e6f4fe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:31.018169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c72f8a3-910c-47c6-8cbc-c09da7e6f4fe None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:31.029771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 622/1257] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:50:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:31.035442 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c009a48e-8613-4b62-b755-f4206cb40b0a None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:31.135858 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c009a48e-8613-4b62-b755-f4206cb40b0a None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:31.168886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac392da0-4519-4ada-b69f-44cc3eb31ff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:31.169539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac392da0-4519-4ada-b69f-44cc3eb31ff2 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:31.170446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac392da0-4519-4ada-b69f-44cc3eb31ff2 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:31.175816 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac392da0-4519-4ada-b69f-44cc3eb31ff2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:31.177145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac392da0-4519-4ada-b69f-44cc3eb31ff2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:31.178573 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac392da0-4519-4ada-b69f-44cc3eb31ff2 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:31.189841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-ac392da0-4519-4ada-b69f-44cc3eb31ff2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:50:31.189841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:31.189841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:31.189841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:31.189841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:31.189841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:31.189841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:31.189841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:31.189841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:31.189841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:31.189841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:31.189841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:31.189841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:50:31.189841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:50:31.189841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:50:31.190842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:50:31.190842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:31.190842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:31.190842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:31.190842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:31.190842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:31.190842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:31.190842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:31.190842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:31.190842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:31.190842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:31.190842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:31.190842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:31.190842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:31.190842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:31.190842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:31.190842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:31.192066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:50:31.192066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:50:31.192066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:50:31.192066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:50:31.192066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:50:31.192066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:50:31.192066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:50:31.192066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:31.192066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 636/1258] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:50:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:31.196749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c72f8a3-910c-47c6-8cbc-c09da7e6f4fe None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:31.300390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c72f8a3-910c-47c6-8cbc-c09da7e6f4fe None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:31.333234 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf54a7b-9ef8-467c-a52d-5dae58f3d8bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:31.333839 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf54a7b-9ef8-467c-a52d-5dae58f3d8bb None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:31.334119 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf54a7b-9ef8-467c-a52d-5dae58f3d8bb None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:31.334392 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf54a7b-9ef8-467c-a52d-5dae58f3d8bb None admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:31.335128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf54a7b-9ef8-467c-a52d-5dae58f3d8bb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:31.335434 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf54a7b-9ef8-467c-a52d-5dae58f3d8bb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:31.336851 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf54a7b-9ef8-467c-a52d-5dae58f3d8bb None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:31.349237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 623/1259] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:50:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:31.355321 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac392da0-4519-4ada-b69f-44cc3eb31ff2 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:31.382114 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:31.460059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac392da0-4519-4ada-b69f-44cc3eb31ff2 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:31.495217 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd38edc-e8df-4a9c-b54b-bce78003ab97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:31.495843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd38edc-e8df-4a9c-b54b-bce78003ab97 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:31.496128 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd38edc-e8df-4a9c-b54b-bce78003ab97 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:31.496400 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd38edc-e8df-4a9c-b54b-bce78003ab97 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:31.497148 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd38edc-e8df-4a9c-b54b-bce78003ab97 None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=b29f444d56ec49308f7fef16f4d1eea5) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:31.497461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd38edc-e8df-4a9c-b54b-bce78003ab97 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:31.500153 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd38edc-e8df-4a9c-b54b-bce78003ab97 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:31.505698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 637/1260] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:50:31 2022] GET /identity/v3/role_assignments?user.id=b29f444d56ec49308f7fef16f4d1eea5&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=19c006412bec45dc8febd7cc86b9678f => generated 259 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:33.382643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d335ae-545a-4343-b89f-120cb2e41f3a None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:33.383229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d335ae-545a-4343-b89f-120cb2e41f3a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:33.383229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d335ae-545a-4343-b89f-120cb2e41f3a None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:33.384503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 624/1261] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:50:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:50:33.390474 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b221e8f9-8ac8-41db-b872-5ba5459b40bd None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:33.390888 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b221e8f9-8ac8-41db-b872-5ba5459b40bd None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:33.391100 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b221e8f9-8ac8-41db-b872-5ba5459b40bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:33.409643 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-b221e8f9-8ac8-41db-b872-5ba5459b40bd None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:50:33.411051 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b221e8f9-8ac8-41db-b872-5ba5459b40bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:50:33.477150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 638/1262] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:50:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:50:33.750237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d335ae-545a-4343-b89f-120cb2e41f3a None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:33.760991 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:33.850272 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d335ae-545a-4343-b89f-120cb2e41f3a None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:33.884118 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eee480b-238d-464e-a7aa-da64d7e30c23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:33.884734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eee480b-238d-464e-a7aa-da64d7e30c23 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:33.885010 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eee480b-238d-464e-a7aa-da64d7e30c23 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:33.885373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eee480b-238d-464e-a7aa-da64d7e30c23 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:33.895766 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eee480b-238d-464e-a7aa-da64d7e30c23 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:50:33.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-9eee480b-238d-464e-a7aa-da64d7e30c23 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:50:33.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:33.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:33.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:33.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:33.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:33.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:33.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:33.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:33.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:33.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:33.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:33.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:50:33.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:50:33.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:50:33.900587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:50:33.901692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:50:33.901692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:50:33.901692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:50:33.901692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:50:33.901692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:33.901692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:33.901692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:33.901692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:33.901692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:33.901692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:33.901692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:33.901692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:33.901692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:33.901692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:33.901692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:33.901692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:33.902702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:33.902702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:33.902702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:33.902702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:33.902702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:50:33.902702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:50:33.902702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:50:33.902702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:50:33.902702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:50:33.902702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:50:33.902702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:50:33.902702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:50:33.902702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 625/1263] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:50:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:50:33.908165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b221e8f9-8ac8-41db-b872-5ba5459b40bd None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:33.927179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:34.007170 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b221e8f9-8ac8-41db-b872-5ba5459b40bd None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:34.040783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32301e2d-eb90-4572-b7b1-9e00df8e6d1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:34.041418 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32301e2d-eb90-4572-b7b1-9e00df8e6d1c None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:34.041699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32301e2d-eb90-4572-b7b1-9e00df8e6d1c None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:34.041972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32301e2d-eb90-4572-b7b1-9e00df8e6d1c None admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:34.042640 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32301e2d-eb90-4572-b7b1-9e00df8e6d1c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:34.042948 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32301e2d-eb90-4572-b7b1-9e00df8e6d1c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:34.044314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32301e2d-eb90-4572-b7b1-9e00df8e6d1c None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:34.053026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 639/1264] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:50:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:34.058981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eee480b-238d-464e-a7aa-da64d7e30c23 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:34.125727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:34.157956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eee480b-238d-464e-a7aa-da64d7e30c23 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:34.190528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05401d5-7799-499b-98f3-229c0a92c662 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:34.191141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05401d5-7799-499b-98f3-229c0a92c662 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:34.191421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05401d5-7799-499b-98f3-229c0a92c662 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:34.191746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05401d5-7799-499b-98f3-229c0a92c662 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:34.202028 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05401d5-7799-499b-98f3-229c0a92c662 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:34.203518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05401d5-7799-499b-98f3-229c0a92c662 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:34.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-f05401d5-7799-499b-98f3-229c0a92c662 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:34.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:34.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:34.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:34.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:34.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:34.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:34.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:34.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:34.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:34.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:34.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:34.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:50:34.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:50:34.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:50:34.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:50:34.218841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:34.218841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:34.218841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:34.218841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:34.218841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:34.218841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:34.218841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:34.218841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:34.218841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:34.218841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:34.218841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:34.218841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:34.218841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:34.218841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:34.218841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:34.218841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:34.218841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:50:34.219944 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:50:34.219944 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:34.219944 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:50:34.219944 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 626/1265] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:34.224826 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32301e2d-eb90-4572-b7b1-9e00df8e6d1c None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:34.308823 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:34.326975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32301e2d-eb90-4572-b7b1-9e00df8e6d1c None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:34.361955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b3e6fa-4fa8-4646-b1a7-8ddc2eb9a8ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:34.362555 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b3e6fa-4fa8-4646-b1a7-8ddc2eb9a8ef None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:34.362834 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b3e6fa-4fa8-4646-b1a7-8ddc2eb9a8ef None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:34.363103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b3e6fa-4fa8-4646-b1a7-8ddc2eb9a8ef None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:34.363739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b3e6fa-4fa8-4646-b1a7-8ddc2eb9a8ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:34.364039 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b3e6fa-4fa8-4646-b1a7-8ddc2eb9a8ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:34.365423 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b3e6fa-4fa8-4646-b1a7-8ddc2eb9a8ef None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:34.376599 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 640/1266] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:50:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:34.382347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05401d5-7799-499b-98f3-229c0a92c662 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:34.483082 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05401d5-7799-499b-98f3-229c0a92c662 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:34.502570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:34.516418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f63f345-9f7d-4eaf-bb35-798f8a7a40ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:34.517043 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f63f345-9f7d-4eaf-bb35-798f8a7a40ca None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:34.517358 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f63f345-9f7d-4eaf-bb35-798f8a7a40ca None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:34.517712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f63f345-9f7d-4eaf-bb35-798f8a7a40ca None admin] PATH_INFO: `/v3/users/placement` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:34.524936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f63f345-9f7d-4eaf-bb35-798f8a7a40ca None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:34.526425 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f63f345-9f7d-4eaf-bb35-798f8a7a40ca None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:34.538188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-5f63f345-9f7d-4eaf-bb35-798f8a7a40ca None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 04 00:50:34.538188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:34.538188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:34.538188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:34.538188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:34.538188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:34.538188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:34.538188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:34.538188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:34.538188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:34.538188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:34.538188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:34.538188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:50:34.538188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:50:34.538188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:50:34.538188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:34.539297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:34.540492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:34.540492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:34.540492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:50:34.540492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:50:34.540492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:50:34.540492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:50:34.540492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:50:34.540492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:50:34.540492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 04 00:50:34.540492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:50:34.540492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 627/1267] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:34 2022] GET /identity/v3/users/placement => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:34.545223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b3e6fa-4fa8-4646-b1a7-8ddc2eb9a8ef None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:34.645875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b3e6fa-4fa8-4646-b1a7-8ddc2eb9a8ef None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:34.678510 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85b9408-5ec9-489f-bb1e-57748375f50b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:34.679104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85b9408-5ec9-489f-bb1e-57748375f50b None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:34.679385 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85b9408-5ec9-489f-bb1e-57748375f50b None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:34.679659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85b9408-5ec9-489f-bb1e-57748375f50b None admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:34.680378 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85b9408-5ec9-489f-bb1e-57748375f50b None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:34.680678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85b9408-5ec9-489f-bb1e-57748375f50b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:34.682135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c85b9408-5ec9-489f-bb1e-57748375f50b None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:34.691358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:34.704187 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 641/1268] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Sep 4 00:50:34 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:34.709933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f63f345-9f7d-4eaf-bb35-798f8a7a40ca None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:34.812163 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f63f345-9f7d-4eaf-bb35-798f8a7a40ca None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:34.845270 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff073c-e8b7-4577-9ae7-d6dbf72f717f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:34.845886 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff073c-e8b7-4577-9ae7-d6dbf72f717f None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:34.846165 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff073c-e8b7-4577-9ae7-d6dbf72f717f None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:34.846436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff073c-e8b7-4577-9ae7-d6dbf72f717f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:34.860548 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ff073c-e8b7-4577-9ae7-d6dbf72f717f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:34.862023 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ff073c-e8b7-4577-9ae7-d6dbf72f717f None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:34.876351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-b7ff073c-e8b7-4577-9ae7-d6dbf72f717f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:34.876351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:34.876351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:34.876351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:34.876351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:34.876351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:34.876351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:34.876351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:34.876351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:34.876351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:34.876351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:34.876351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:34.876351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:50:34.876351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:50:34.876351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:50:34.876351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:50:34.877506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:34.877506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:34.877506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:34.877506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:34.877506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:34.877506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:34.877506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:34.877506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:34.877506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:34.877506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:34.877506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:34.877506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:34.877506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:34.877506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:34.877506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:34.877506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:34.877506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:50:34.878760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:50:34.878760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:34.878760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:50:34.878760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 628/1269] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:34.883363 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85b9408-5ec9-489f-bb1e-57748375f50b None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:34.980881 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85b9408-5ec9-489f-bb1e-57748375f50b None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:35.015571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fc072b-2daf-4c0d-a406-dd4e864c37ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:35.016171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fc072b-2daf-4c0d-a406-dd4e864c37ff None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:35.016449 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fc072b-2daf-4c0d-a406-dd4e864c37ff None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:35.016720 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fc072b-2daf-4c0d-a406-dd4e864c37ff None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:35.017388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fc072b-2daf-4c0d-a406-dd4e864c37ff None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:35.017691 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fc072b-2daf-4c0d-a406-dd4e864c37ff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:35.019049 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fc072b-2daf-4c0d-a406-dd4e864c37ff None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:35.030384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 642/1270] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:50:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:35.036136 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff073c-e8b7-4577-9ae7-d6dbf72f717f None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:35.067711 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:35.139464 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff073c-e8b7-4577-9ae7-d6dbf72f717f None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:35.172794 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e037610-c91e-4783-840f-a856d5227493 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:35.173441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e037610-c91e-4783-840f-a856d5227493 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:35.173719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e037610-c91e-4783-840f-a856d5227493 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:35.174009 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e037610-c91e-4783-840f-a856d5227493 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:35.180340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e037610-c91e-4783-840f-a856d5227493 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:35.181791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e037610-c91e-4783-840f-a856d5227493 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:35.192681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-4e037610-c91e-4783-840f-a856d5227493 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:50:35.192681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:35.192681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:35.192681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:35.192681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:35.192681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:35.192681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:35.192681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:35.192681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:35.192681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:35.192681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:35.192681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:35.192681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:50:35.192681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:50:35.192681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:50:35.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:50:35.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:35.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:35.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:35.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:35.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:35.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:35.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:35.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:35.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:35.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:35.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:35.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:35.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:35.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:35.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:35.193749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:35.194845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:50:35.194845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:50:35.194845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:50:35.194845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:50:35.194845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:50:35.194845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:50:35.194845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:50:35.194845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:50:35.194845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 629/1271] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:50:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:35.199620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fc072b-2daf-4c0d-a406-dd4e864c37ff None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:35.257444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:35.299019 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fc072b-2daf-4c0d-a406-dd4e864c37ff None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:35.332412 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1488988-4cd3-473e-85df-177e820c20e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:35.333009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1488988-4cd3-473e-85df-177e820c20e5 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:35.333317 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1488988-4cd3-473e-85df-177e820c20e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:35.333587 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1488988-4cd3-473e-85df-177e820c20e5 None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:35.334318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1488988-4cd3-473e-85df-177e820c20e5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:35.334617 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1488988-4cd3-473e-85df-177e820c20e5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:35.336134 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1488988-4cd3-473e-85df-177e820c20e5 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:35.348428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 643/1272] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:50:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:35.354474 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e037610-c91e-4783-840f-a856d5227493 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:35.447365 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:35.456566 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e037610-c91e-4783-840f-a856d5227493 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:35.490762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b56f72-66f6-46f7-ab66-79d0b59c61f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:35.491419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b56f72-66f6-46f7-ab66-79d0b59c61f4 None admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:35.491696 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b56f72-66f6-46f7-ab66-79d0b59c61f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:35.491968 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b56f72-66f6-46f7-ab66-79d0b59c61f4 None admin] PATH_INFO: `/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/b29f444d56ec49308f7fef16f4d1eea5/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:35.493866 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b56f72-66f6-46f7-ab66-79d0b59c61f4 None admin] RBAC: Authorizing `identity:create_grant(project_id=b1f56314d44d4faaa88e735ea196783f, user_id=b29f444d56ec49308f7fef16f4d1eea5, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:35.495323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b56f72-66f6-46f7-ab66-79d0b59c61f4 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:35.569986 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 630/1273] 10.222.0.42 () {58 vars in 1439 bytes} [Sun Sep 4 00:50:35 2022] PUT /identity/v3/projects/b1f56314d44d4faaa88e735ea196783f/users/b29f444d56ec49308f7fef16f4d1eea5/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 00:50:37.443321 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb74932-1a19-46e1-be22-4c7676cdc29d None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:37.443860 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb74932-1a19-46e1-be22-4c7676cdc29d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:37.444239 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb74932-1a19-46e1-be22-4c7676cdc29d None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:37.446712 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 644/1274] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:50:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:50:37.452110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5484c667-689c-4dbe-a39c-8e6c1230df30 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:37.452515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5484c667-689c-4dbe-a39c-8e6c1230df30 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:37.452650 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5484c667-689c-4dbe-a39c-8e6c1230df30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:37.471340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-5484c667-689c-4dbe-a39c-8e6c1230df30 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:50:37.472572 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-5484c667-689c-4dbe-a39c-8e6c1230df30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:50:37.546027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 631/1275] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:50:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:50:37.814332 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb74932-1a19-46e1-be22-4c7676cdc29d None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:37.925032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb74932-1a19-46e1-be22-4c7676cdc29d None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:37.942466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:37.958994 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482c39a7-348c-45db-8ca1-e06f557008ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:37.959671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482c39a7-348c-45db-8ca1-e06f557008ed None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:37.959958 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482c39a7-348c-45db-8ca1-e06f557008ed None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:37.960223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482c39a7-348c-45db-8ca1-e06f557008ed None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:37.970799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482c39a7-348c-45db-8ca1-e06f557008ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 04 00:50:37.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-482c39a7-348c-45db-8ca1-e06f557008ed None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:50:37.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:37.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:37.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:37.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:37.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:37.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:37.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:37.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:37.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:37.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:37.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:37.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 04 00:50:37.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 04 00:50:37.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 04 00:50:37.975563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 04 00:50:37.976701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 04 00:50:37.976701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 04 00:50:37.976701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 04 00:50:37.976701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 04 00:50:37.976701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:37.976701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:37.976701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:37.976701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:37.976701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:37.976701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:37.976701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:37.976701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:37.976701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:37.976701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:37.976701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:37.976701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:37.977763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:37.977763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:37.977763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:37.977763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:37.977763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 04 00:50:37.977763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 04 00:50:37.977763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 04 00:50:37.977763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 04 00:50:37.977763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 04 00:50:37.977763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 04 00:50:37.977763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 04 00:50:37.977763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:37.977763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 645/1276] 10.222.0.42 () {56 vars in 1096 bytes} [Sun Sep 4 00:50:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:37.983400 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5484c667-689c-4dbe-a39c-8e6c1230df30 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:38.064020 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:38.081479 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5484c667-689c-4dbe-a39c-8e6c1230df30 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:38.115169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee506735-248c-4399-9007-41c453c939fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:38.115766 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee506735-248c-4399-9007-41c453c939fb None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:38.116048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee506735-248c-4399-9007-41c453c939fb None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:38.116324 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee506735-248c-4399-9007-41c453c939fb None admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:38.116987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee506735-248c-4399-9007-41c453c939fb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:38.117370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee506735-248c-4399-9007-41c453c939fb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:38.118737 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee506735-248c-4399-9007-41c453c939fb None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:38.127072 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 632/1277] 10.222.0.42 () {56 vars in 1099 bytes} [Sun Sep 4 00:50:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:38.132788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482c39a7-348c-45db-8ca1-e06f557008ed None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:38.236386 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482c39a7-348c-45db-8ca1-e06f557008ed None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:38.274636 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a59cad-dd48-47a7-88a4-03148cd676e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:38.275297 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a59cad-dd48-47a7-88a4-03148cd676e2 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:38.275579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a59cad-dd48-47a7-88a4-03148cd676e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:38.275851 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a59cad-dd48-47a7-88a4-03148cd676e2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:38.288187 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a59cad-dd48-47a7-88a4-03148cd676e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:38.289770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a59cad-dd48-47a7-88a4-03148cd676e2 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:38.306040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-14a59cad-dd48-47a7-88a4-03148cd676e2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:38.306040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:38.306040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:38.306040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:38.306040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:38.306040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:38.306040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:38.306040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:38.306040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:38.306040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:38.306040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:38.306040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:38.306040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:50:38.306040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:50:38.306040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:50:38.306040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:50:38.311515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:38.311515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:38.311515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:38.311515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:38.311515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:38.311515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:38.311515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:38.311515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:38.311515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:38.311515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:38.311515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:38.311515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:38.311515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:38.311515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:38.311515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:38.311515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:38.311515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:50:38.312525 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:50:38.312525 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:38.312525 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:38.312525 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 646/1278] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:38.313543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee506735-248c-4399-9007-41c453c939fb None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:38.428832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee506735-248c-4399-9007-41c453c939fb None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:38.467755 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f9cd64-42bd-4e69-8571-4fc39b6b43ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:38.468428 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f9cd64-42bd-4e69-8571-4fc39b6b43ac None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:38.468696 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f9cd64-42bd-4e69-8571-4fc39b6b43ac None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:38.468974 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f9cd64-42bd-4e69-8571-4fc39b6b43ac None admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:38.469679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f9cd64-42bd-4e69-8571-4fc39b6b43ac None admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:38.469990 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f9cd64-42bd-4e69-8571-4fc39b6b43ac None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:38.471383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f9cd64-42bd-4e69-8571-4fc39b6b43ac None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:38.484384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 633/1279] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:50:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:50:38.490264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a59cad-dd48-47a7-88a4-03148cd676e2 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:38.524657 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:38.596611 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a59cad-dd48-47a7-88a4-03148cd676e2 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:38.631957 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a1b86a-886b-45cc-ba1f-414098bb28ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:38.632754 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a1b86a-886b-45cc-ba1f-414098bb28ed None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:38.633168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a1b86a-886b-45cc-ba1f-414098bb28ed None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:38.633443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a1b86a-886b-45cc-ba1f-414098bb28ed None admin] PATH_INFO: `/v3/users/placement` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:38.641271 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a1b86a-886b-45cc-ba1f-414098bb28ed None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:38.642707 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a1b86a-886b-45cc-ba1f-414098bb28ed None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:38.654785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-d8a1b86a-886b-45cc-ba1f-414098bb28ed None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 04 00:50:38.654785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:38.654785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:38.654785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:38.654785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:38.654785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:38.654785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:38.654785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:38.654785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:38.654785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:38.654785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:38.654785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:38.654785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 04 00:50:38.654785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 04 00:50:38.654785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 04 00:50:38.654785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:38.655955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:38.657219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:38.657219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:38.657219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 04 00:50:38.657219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 04 00:50:38.657219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 04 00:50:38.657219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 04 00:50:38.657219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 04 00:50:38.657219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 04 00:50:38.657219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 04 00:50:38.657219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:38.657219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 647/1280] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:38 2022] GET /identity/v3/users/placement => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:38.662083 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f9cd64-42bd-4e69-8571-4fc39b6b43ac None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:38.666118 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:38.766790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f9cd64-42bd-4e69-8571-4fc39b6b43ac None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:38.798618 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b78345-683c-4736-8250-143102c6304c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:38.799210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b78345-683c-4736-8250-143102c6304c None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:38.799542 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b78345-683c-4736-8250-143102c6304c None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:38.799816 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b78345-683c-4736-8250-143102c6304c None admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:38.800534 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b78345-683c-4736-8250-143102c6304c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:38.800836 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b78345-683c-4736-8250-143102c6304c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:38.802276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b78345-683c-4736-8250-143102c6304c None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:38.820512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 634/1281] 10.222.0.42 () {56 vars in 1143 bytes} [Sun Sep 4 00:50:38 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:38.826982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a1b86a-886b-45cc-ba1f-414098bb28ed None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:38.917205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:38.927058 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a1b86a-886b-45cc-ba1f-414098bb28ed None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:38.959981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ff5c67-522e-456b-b8f9-bc864b966e06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:38.960579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ff5c67-522e-456b-b8f9-bc864b966e06 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:38.960970 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ff5c67-522e-456b-b8f9-bc864b966e06 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:38.961266 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ff5c67-522e-456b-b8f9-bc864b966e06 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:39.026843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ff5c67-522e-456b-b8f9-bc864b966e06 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:39.028223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ff5c67-522e-456b-b8f9-bc864b966e06 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:39.114209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:39.125121 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 648/1282] 10.222.0.42 () {58 vars in 1282 bytes} [Sun Sep 4 00:50:38 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 04 00:50:39.131021 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b78345-683c-4736-8250-143102c6304c None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:39.191317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:39.228895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b78345-683c-4736-8250-143102c6304c None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:39.261662 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a6448a-157a-4198-9e48-5c1e6ad63736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:39.262277 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a6448a-157a-4198-9e48-5c1e6ad63736 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:39.262556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a6448a-157a-4198-9e48-5c1e6ad63736 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:39.262823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a6448a-157a-4198-9e48-5c1e6ad63736 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:39.273129 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a6448a-157a-4198-9e48-5c1e6ad63736 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:39.274605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a6448a-157a-4198-9e48-5c1e6ad63736 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:39.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-c9a6448a-157a-4198-9e48-5c1e6ad63736 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:39.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:39.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:39.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:39.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:39.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:39.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:39.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:39.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:39.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:39.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:39.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:39.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 04 00:50:39.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 04 00:50:39.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 04 00:50:39.288773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 04 00:50:39.289897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:39.289897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:39.289897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:39.289897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:39.289897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:39.289897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:39.289897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:39.289897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:39.289897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:39.289897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:39.289897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:39.289897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:39.289897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:39.289897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:39.289897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:39.289897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:39.289897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 04 00:50:39.290936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 04 00:50:39.290936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 04 00:50:39.290936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:50:39.290936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 635/1283] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:50:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:39.296488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ff5c67-522e-456b-b8f9-bc864b966e06 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:39.394839 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ff5c67-522e-456b-b8f9-bc864b966e06 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:39.429696 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6255cc9-fc39-4b32-9cad-62cf5ed5db0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:39.430308 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6255cc9-fc39-4b32-9cad-62cf5ed5db0d None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:39.430592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6255cc9-fc39-4b32-9cad-62cf5ed5db0d None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:39.430863 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6255cc9-fc39-4b32-9cad-62cf5ed5db0d None admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:39.431506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6255cc9-fc39-4b32-9cad-62cf5ed5db0d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:39.431813 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6255cc9-fc39-4b32-9cad-62cf5ed5db0d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:39.433213 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6255cc9-fc39-4b32-9cad-62cf5ed5db0d None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:39.444272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 649/1284] 10.222.0.42 () {56 vars in 1109 bytes} [Sun Sep 4 00:50:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 00:50:39.450037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a6448a-157a-4198-9e48-5c1e6ad63736 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:39.551347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a6448a-157a-4198-9e48-5c1e6ad63736 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:39.564711 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:39.583805 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598033da-6643-4df0-b83a-311eca93f5d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:39.584424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598033da-6643-4df0-b83a-311eca93f5d6 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:39.584702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598033da-6643-4df0-b83a-311eca93f5d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:39.584972 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598033da-6643-4df0-b83a-311eca93f5d6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:39.592019 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598033da-6643-4df0-b83a-311eca93f5d6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:39.593473 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598033da-6643-4df0-b83a-311eca93f5d6 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:39.604416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application [None req-598033da-6643-4df0-b83a-311eca93f5d6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:50:39.604416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:39.604416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:39.604416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:39.604416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:39.604416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:39.604416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:39.604416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:39.604416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:39.604416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:39.604416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:39.604416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:39.604416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 04 00:50:39.604416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 04 00:50:39.604416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 04 00:50:39.605471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 04 00:50:39.605471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:39.605471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:39.605471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:39.605471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:39.605471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:39.605471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:39.605471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:39.605471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:39.605471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:39.605471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:39.605471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:39.605471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:39.605471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:39.605471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:39.605471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:39.605471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:39.606390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 04 00:50:39.606390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 04 00:50:39.606390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 04 00:50:39.606390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 04 00:50:39.606390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 04 00:50:39.606390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 04 00:50:39.606390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 04 00:50:39.606390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: ERROR keystone.server.flask.application Sep 04 00:50:39.606390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 636/1285] 10.222.0.42 () {56 vars in 1111 bytes} [Sun Sep 4 00:50:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 04 00:50:39.612480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6255cc9-fc39-4b32-9cad-62cf5ed5db0d None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:39.662920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:39.712147 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6255cc9-fc39-4b32-9cad-62cf5ed5db0d None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:39.745493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2573f9-1309-4f77-9863-467b27e1f32f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:39.746136 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2573f9-1309-4f77-9863-467b27e1f32f None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:39.746411 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2573f9-1309-4f77-9863-467b27e1f32f None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:39.746685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2573f9-1309-4f77-9863-467b27e1f32f None admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:39.747429 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2573f9-1309-4f77-9863-467b27e1f32f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:39.748226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2573f9-1309-4f77-9863-467b27e1f32f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:39.749684 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2573f9-1309-4f77-9863-467b27e1f32f None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:39.761324 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 650/1286] 10.222.0.42 () {56 vars in 1148 bytes} [Sun Sep 4 00:50:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:39.768171 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598033da-6643-4df0-b83a-311eca93f5d6 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:39.866869 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598033da-6643-4df0-b83a-311eca93f5d6 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:39.900200 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdec1da7-6a61-4405-b34a-7077d041d215 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:39.900805 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdec1da7-6a61-4405-b34a-7077d041d215 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:39.901134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdec1da7-6a61-4405-b34a-7077d041d215 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:39.901440 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdec1da7-6a61-4405-b34a-7077d041d215 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:39.902165 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdec1da7-6a61-4405-b34a-7077d041d215 None admin] RBAC: Adding query filter params (role.id=19c006412bec45dc8febd7cc86b9678f, scope.project.id=b1f56314d44d4faaa88e735ea196783f, user.id=b29f444d56ec49308f7fef16f4d1eea5) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:39.902473 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdec1da7-6a61-4405-b34a-7077d041d215 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:39.904759 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdec1da7-6a61-4405-b34a-7077d041d215 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:39.910474 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 637/1287] 10.222.0.42 () {56 vars in 1374 bytes} [Sun Sep 4 00:50:39 2022] GET /identity/v3/role_assignments?user.id=b29f444d56ec49308f7fef16f4d1eea5&scope.project.id=b1f56314d44d4faaa88e735ea196783f&role.id=19c006412bec45dc8febd7cc86b9678f => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:41.862896 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b165ab-8e30-453a-96e9-f049ee8a9ccb None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:41.863452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b165ab-8e30-453a-96e9-f049ee8a9ccb None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:41.863452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b165ab-8e30-453a-96e9-f049ee8a9ccb None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:41.864772 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 651/1288] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:50:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:50:41.869877 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e3f627-8fed-43d8-9b4e-6fc17ae1ce91 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:41.870286 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e3f627-8fed-43d8-9b4e-6fc17ae1ce91 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:41.870438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e3f627-8fed-43d8-9b4e-6fc17ae1ce91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:41.889090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-b0e3f627-8fed-43d8-9b4e-6fc17ae1ce91 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:50:41.890278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b0e3f627-8fed-43d8-9b4e-6fc17ae1ce91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:50:41.918965 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:41.955986 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 638/1289] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:50:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 04 00:50:42.232517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b165ab-8e30-453a-96e9-f049ee8a9ccb None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:42.330949 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b165ab-8e30-453a-96e9-f049ee8a9ccb None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:42.341733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:42.364799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376a0a8c-5068-4abf-83bf-2d1f21075c96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:42.365447 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376a0a8c-5068-4abf-83bf-2d1f21075c96 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:42.365727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376a0a8c-5068-4abf-83bf-2d1f21075c96 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:42.365997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376a0a8c-5068-4abf-83bf-2d1f21075c96 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:42.366576 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376a0a8c-5068-4abf-83bf-2d1f21075c96 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:42.367933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376a0a8c-5068-4abf-83bf-2d1f21075c96 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:42.377291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-376a0a8c-5068-4abf-83bf-2d1f21075c96 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 04 00:50:42.377291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:42.377291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:50:42.377291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:50:42.377291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:50:42.377291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:50:42.377291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:50:42.377291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:50:42.377291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 04 00:50:42.377291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:42.377291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:50:42.377291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:42.377291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:42.377291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:42.377291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:42.377291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:42.377291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:42.378534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:42.378534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:42.378534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:42.378534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:42.378534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:42.378534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:42.378534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:50:42.378534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:50:42.378534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:50:42.378534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:50:42.378534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:42.378534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:42.378534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:42.378534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:42.378534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:42.378534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:42.379574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:42.379574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:42.379574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:42.379574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:42.379574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:42.379574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:42.379574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:42.379574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:42.379574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:42.379574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:42.379574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:50:42.379574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:50:42.379574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 04 00:50:42.379574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:42.379574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 652/1290] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 4 00:50:42 2022] GET /identity/v3/services/placement => generated 90 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:42.384730 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e3f627-8fed-43d8-9b4e-6fc17ae1ce91 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:42.488453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e3f627-8fed-43d8-9b4e-6fc17ae1ce91 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:42.521616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fac946-3c1f-453f-935e-cc1560f99e20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:42.522213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fac946-3c1f-453f-935e-cc1560f99e20 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:42.522491 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fac946-3c1f-453f-935e-cc1560f99e20 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:42.522759 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fac946-3c1f-453f-935e-cc1560f99e20 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:42.523414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fac946-3c1f-453f-935e-cc1560f99e20 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:42.523713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fac946-3c1f-453f-935e-cc1560f99e20 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:42.525050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fac946-3c1f-453f-935e-cc1560f99e20 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:42.530684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 639/1291] 10.222.0.42 () {56 vars in 1116 bytes} [Sun Sep 4 00:50:42 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:42.536474 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376a0a8c-5068-4abf-83bf-2d1f21075c96 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:42.636386 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376a0a8c-5068-4abf-83bf-2d1f21075c96 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:42.669713 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae496bc-ba28-479e-9eef-893425b63dd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:42.670404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae496bc-ba28-479e-9eef-893425b63dd7 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:42.670682 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae496bc-ba28-479e-9eef-893425b63dd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:42.670952 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae496bc-ba28-479e-9eef-893425b63dd7 None admin] PATH_INFO: `/v3/services` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:42.671568 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae496bc-ba28-479e-9eef-893425b63dd7 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:42.671868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae496bc-ba28-479e-9eef-893425b63dd7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:42.673253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae496bc-ba28-479e-9eef-893425b63dd7 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:42.677692 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 653/1292] 10.222.0.42 () {56 vars in 1116 bytes} [Sun Sep 4 00:50:42 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:44.609281 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b87b09-72ab-427e-827f-bcdc565952c7 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:44.609842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b87b09-72ab-427e-827f-bcdc565952c7 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:44.609842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b87b09-72ab-427e-827f-bcdc565952c7 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:44.611151 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 640/1293] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:50:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:50:44.616319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba0c26e-336c-4b9c-8e2e-1079aecda827 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:44.616715 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba0c26e-336c-4b9c-8e2e-1079aecda827 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:44.616892 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba0c26e-336c-4b9c-8e2e-1079aecda827 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:44.635470 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-4ba0c26e-336c-4b9c-8e2e-1079aecda827 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:50:44.637323 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4ba0c26e-336c-4b9c-8e2e-1079aecda827 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:50:44.665831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:44.703106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 654/1294] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:50:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:50:44.972266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b87b09-72ab-427e-827f-bcdc565952c7 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:45.003747 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:45.073331 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b87b09-72ab-427e-827f-bcdc565952c7 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:45.107280 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad9c993-0b1f-478f-aae6-dddc92d6b056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:45.107882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad9c993-0b1f-478f-aae6-dddc92d6b056 None admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:45.108167 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad9c993-0b1f-478f-aae6-dddc92d6b056 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:45.108440 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad9c993-0b1f-478f-aae6-dddc92d6b056 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:45.109114 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad9c993-0b1f-478f-aae6-dddc92d6b056 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:45.110472 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad9c993-0b1f-478f-aae6-dddc92d6b056 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:45.164324 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 641/1295] 10.222.0.42 () {60 vars in 1141 bytes} [Sun Sep 4 00:50:44 2022] POST /identity/v3/services => generated 246 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:50:47.073572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8cac82-5913-4b8d-9049-c4cb5c3bd6ae None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:47.074119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8cac82-5913-4b8d-9049-c4cb5c3bd6ae None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:47.074119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8cac82-5913-4b8d-9049-c4cb5c3bd6ae None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:47.076230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 655/1296] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:50:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:50:47.081376 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a254d2ac-681c-44ad-9464-0b0bd9b647ac None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:47.081780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a254d2ac-681c-44ad-9464-0b0bd9b647ac None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:47.081933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a254d2ac-681c-44ad-9464-0b0bd9b647ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:47.099861 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-a254d2ac-681c-44ad-9464-0b0bd9b647ac None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:50:47.101045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a254d2ac-681c-44ad-9464-0b0bd9b647ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:50:47.173242 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 642/1297] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:50:47 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:50:47.457278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8cac82-5913-4b8d-9049-c4cb5c3bd6ae None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:47.557509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8cac82-5913-4b8d-9049-c4cb5c3bd6ae None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:47.591321 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbaf7879-be07-4b6f-9fb7-b3f67c5d6993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:47.592044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbaf7879-be07-4b6f-9fb7-b3f67c5d6993 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:47.592317 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbaf7879-be07-4b6f-9fb7-b3f67c5d6993 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:47.592586 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbaf7879-be07-4b6f-9fb7-b3f67c5d6993 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:47.593189 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbaf7879-be07-4b6f-9fb7-b3f67c5d6993 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:47.594534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbaf7879-be07-4b6f-9fb7-b3f67c5d6993 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:47.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-dbaf7879-be07-4b6f-9fb7-b3f67c5d6993 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 04 00:50:47.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:47.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:50:47.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:50:47.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:50:47.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:50:47.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:50:47.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:50:47.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 04 00:50:47.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:47.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:50:47.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:47.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:47.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:47.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:47.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:47.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:47.604488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:47.604488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:47.604488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:47.604488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:47.604488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:47.604488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:47.604488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:50:47.604488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:50:47.604488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:50:47.604488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:50:47.604488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:47.604488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:47.604488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:47.604488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:47.604488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:47.604488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:47.605652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:47.605652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:47.605652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:47.605652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:47.605652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:47.605652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:47.605652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:47.605652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:47.605652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:47.605652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:47.605652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:50:47.605652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:50:47.605652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 04 00:50:47.605652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:47.605652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 656/1298] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 4 00:50:47 2022] GET /identity/v3/services/placement => generated 90 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:47.610896 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a254d2ac-681c-44ad-9464-0b0bd9b647ac None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:47.629358 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:47.707757 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a254d2ac-681c-44ad-9464-0b0bd9b647ac None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:47.741788 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e637e71b-5e34-40ad-bb67-26b7684bd8f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:47.742384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e637e71b-5e34-40ad-bb67-26b7684bd8f8 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:47.742661 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e637e71b-5e34-40ad-bb67-26b7684bd8f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:47.742935 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e637e71b-5e34-40ad-bb67-26b7684bd8f8 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:47.743565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e637e71b-5e34-40ad-bb67-26b7684bd8f8 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:47.743865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e637e71b-5e34-40ad-bb67-26b7684bd8f8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:47.745242 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e637e71b-5e34-40ad-bb67-26b7684bd8f8 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:47.750605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 643/1299] 10.222.0.42 () {56 vars in 1116 bytes} [Sun Sep 4 00:50:47 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:47.756604 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbaf7879-be07-4b6f-9fb7-b3f67c5d6993 None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:47.788388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:47.856502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbaf7879-be07-4b6f-9fb7-b3f67c5d6993 None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:47.889629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43527094-c8c2-4256-8d55-e405b2e595c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:47.890219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43527094-c8c2-4256-8d55-e405b2e595c6 None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:47.890491 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43527094-c8c2-4256-8d55-e405b2e595c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:47.890763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43527094-c8c2-4256-8d55-e405b2e595c6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:47.891455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43527094-c8c2-4256-8d55-e405b2e595c6 None admin] RBAC: Adding query filter params (interface=public, service_id=d951333c27ca480388dfa50b310d7288, region_id=RegionOne) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:47.891758 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43527094-c8c2-4256-8d55-e405b2e595c6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:47.893128 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43527094-c8c2-4256-8d55-e405b2e595c6 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:47.898446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 657/1300] 10.222.0.42 () {56 vars in 1251 bytes} [Sun Sep 4 00:50:47 2022] GET /identity/v3/endpoints?service_id=d951333c27ca480388dfa50b310d7288&interface=public®ion_id=RegionOne => generated 194 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:47.904835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e637e71b-5e34-40ad-bb67-26b7684bd8f8 None admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:47.990419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:48.011049 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e637e71b-5e34-40ad-bb67-26b7684bd8f8 None admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:48.046855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2363e85-ab72-4e77-856e-6ea2f932020a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:48.047469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2363e85-ab72-4e77-856e-6ea2f932020a None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:48.047753 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2363e85-ab72-4e77-856e-6ea2f932020a None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:48.048124 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2363e85-ab72-4e77-856e-6ea2f932020a None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:48.048739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2363e85-ab72-4e77-856e-6ea2f932020a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:48.050225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2363e85-ab72-4e77-856e-6ea2f932020a None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:48.057037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 644/1301] 10.222.0.42 () {56 vars in 1087 bytes} [Sun Sep 4 00:50:47 2022] GET /identity/v3/services => generated 2191 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 00:50:49.942613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcb8214-3400-4131-92d9-920741d9a595 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:49.943163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcb8214-3400-4131-92d9-920741d9a595 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:49.943163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdcb8214-3400-4131-92d9-920741d9a595 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:49.944524 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 658/1302] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:50:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:50:49.949745 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5146a0ee-b2c0-4209-b434-f93368c8158e None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:49.950235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5146a0ee-b2c0-4209-b434-f93368c8158e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:49.950378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5146a0ee-b2c0-4209-b434-f93368c8158e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:49.968691 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-5146a0ee-b2c0-4209-b434-f93368c8158e None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:50:49.969900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-5146a0ee-b2c0-4209-b434-f93368c8158e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:50:50.034696 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 645/1303] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:50:49 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 04 00:50:50.301305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdcb8214-3400-4131-92d9-920741d9a595 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:50.394852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:50.401355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdcb8214-3400-4131-92d9-920741d9a595 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:50.434505 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1159d4c7-7bf9-4145-999f-dee446809b8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:50.435129 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1159d4c7-7bf9-4145-999f-dee446809b8a None admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:50.435404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1159d4c7-7bf9-4145-999f-dee446809b8a None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:50.435673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1159d4c7-7bf9-4145-999f-dee446809b8a None admin] PATH_INFO: `/v3/services/placement` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:50.436245 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1159d4c7-7bf9-4145-999f-dee446809b8a None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:50.437676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1159d4c7-7bf9-4145-999f-dee446809b8a None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:50.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application [None req-1159d4c7-7bf9-4145-999f-dee446809b8a None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 04 00:50:50.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:50.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 04 00:50:50.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 04 00:50:50.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 04 00:50:50.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 04 00:50:50.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 04 00:50:50.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:50:50.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 04 00:50:50.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:50.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 04 00:50:50.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:50.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 04 00:50:50.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 04 00:50:50.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 04 00:50:50.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 04 00:50:50.446106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 04 00:50:50.447226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 04 00:50:50.447226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 04 00:50:50.447226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 04 00:50:50.447226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 04 00:50:50.447226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 04 00:50:50.447226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 04 00:50:50.447226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 04 00:50:50.447226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 04 00:50:50.447226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 04 00:50:50.447226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 04 00:50:50.447226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 04 00:50:50.447226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 04 00:50:50.447226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 04 00:50:50.447226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 04 00:50:50.447226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 04 00:50:50.447226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.get_or_create( Sep 04 00:50:50.448235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 04 00:50:50.448235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application with Lock( Sep 04 00:50:50.448235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 04 00:50:50.448235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self._enter() Sep 04 00:50:50.448235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 04 00:50:50.448235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 04 00:50:50.448235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 04 00:50:50.448235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application return self.creator() Sep 04 00:50:50.448235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 04 00:50:50.448235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application created_value = creator( Sep 04 00:50:50.448235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 04 00:50:50.448235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 04 00:50:50.448235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 04 00:50:50.448235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: ERROR keystone.server.flask.application Sep 04 00:50:50.448235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 659/1304] 10.222.0.42 () {56 vars in 1117 bytes} [Sun Sep 4 00:50:50 2022] GET /identity/v3/services/placement => generated 90 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 04 00:50:50.453868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5146a0ee-b2c0-4209-b434-f93368c8158e None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:50.532058 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:50:50.554933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5146a0ee-b2c0-4209-b434-f93368c8158e None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:50.589714 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f122a169-884d-47fb-b7f2-81936c907d79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:50.590316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f122a169-884d-47fb-b7f2-81936c907d79 None admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:50.590597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f122a169-884d-47fb-b7f2-81936c907d79 None admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:50.590903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f122a169-884d-47fb-b7f2-81936c907d79 None admin] PATH_INFO: `/v3/services` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:50.591541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f122a169-884d-47fb-b7f2-81936c907d79 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 00:50:50.591840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f122a169-884d-47fb-b7f2-81936c907d79 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:50.593251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f122a169-884d-47fb-b7f2-81936c907d79 None admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:50.598259 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 646/1305] 10.222.0.42 () {56 vars in 1116 bytes} [Sun Sep 4 00:50:50 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 00:50:50.604226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1159d4c7-7bf9-4145-999f-dee446809b8a None admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:50:50.703777 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1159d4c7-7bf9-4145-999f-dee446809b8a None admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:50:50.737631 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8968358-4725-45f5-b5f0-6fa1ad735c12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:50:50.738223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8968358-4725-45f5-b5f0-6fa1ad735c12 None admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:50:50.738500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8968358-4725-45f5-b5f0-6fa1ad735c12 None admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:50:50.738770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8968358-4725-45f5-b5f0-6fa1ad735c12 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:50:50.739475 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8968358-4725-45f5-b5f0-6fa1ad735c12 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:50:50.740827 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8968358-4725-45f5-b5f0-6fa1ad735c12 None admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:50:50.879997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 660/1306] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 00:50:50 2022] POST /identity/v3/endpoints => generated 331 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 00:51:00.595252 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=30 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:51:00.875488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:33.767267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f64d2f-5936-4145-9d5d-b0b76c21dbca None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:33.767697 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f64d2f-5936-4145-9d5d-b0b76c21dbca None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:33.767834 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f64d2f-5936-4145-9d5d-b0b76c21dbca None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:33.774503 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95a0c68-92be-448a-9a1e-6740300bcd65 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:33.774931 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95a0c68-92be-448a-9a1e-6740300bcd65 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:33.775188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95a0c68-92be-448a-9a1e-6740300bcd65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:33.780292 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 647/1307] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:56:33 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:56:33.909070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-b95a0c68-92be-448a-9a1e-6740300bcd65 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:56:33.910962 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b95a0c68-92be-448a-9a1e-6740300bcd65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:34.114467 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 661/1308] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:56:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 341 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 00:56:34.435796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6efd9e-ed55-49c7-a8c2-775f894f9aeb None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:34.436293 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6efd9e-ed55-49c7-a8c2-775f894f9aeb None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:34.436832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6efd9e-ed55-49c7-a8c2-775f894f9aeb None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:34.438684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 648/1309] 10.222.0.42 () {54 vars in 965 bytes} [Sun Sep 4 00:56:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:56:34.445189 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2549ebdf-a125-413e-ae00-587bf6e71081 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:34.445483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2549ebdf-a125-413e-ae00-587bf6e71081 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:34.445759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2549ebdf-a125-413e-ae00-587bf6e71081 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:34.541796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-2549ebdf-a125-413e-ae00-587bf6e71081 None None] MFA Rules not processed for user `da4f44b86ca94adf959d9a41d915c902`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:56:34.543051 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-2549ebdf-a125-413e-ae00-587bf6e71081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:34.568359 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:34.644426 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 662/1310] 10.222.0.42 () {58 vars in 1062 bytes} [Sun Sep 4 00:56:34 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:56:34.655200 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6efd9e-ed55-49c7-a8c2-775f894f9aeb None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:56:34.657323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ca6efd9e-ed55-49c7-a8c2-775f894f9aeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:34.796337 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6efd9e-ed55-49c7-a8c2-775f894f9aeb None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:56:34.886061 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ee9555-1a7e-4336-bc59-6bf4efc0e014 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:56:34.887343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ee9555-1a7e-4336-bc59-6bf4efc0e014 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:34.887843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ee9555-1a7e-4336-bc59-6bf4efc0e014 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:34.888327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ee9555-1a7e-4336-bc59-6bf4efc0e014 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:34.890236 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-27ee9555-1a7e-4336-bc59-6bf4efc0e014 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:34.953481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:35.017922 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ee9555-1a7e-4336-bc59-6bf4efc0e014 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:56:35.019714 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ee9555-1a7e-4336-bc59-6bf4efc0e014 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:56:35.042651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 649/1311] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:56:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:56:37.330609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfb19b7-d387-4593-ad7e-b89e8ec76563 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:37.331158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfb19b7-d387-4593-ad7e-b89e8ec76563 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:37.331158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfb19b7-d387-4593-ad7e-b89e8ec76563 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:37.332517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 663/1312] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:56:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:56:37.338625 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408bf464-7ffe-4a4e-989d-6dfe4a6cae70 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:37.339052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408bf464-7ffe-4a4e-989d-6dfe4a6cae70 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:37.339321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408bf464-7ffe-4a4e-989d-6dfe4a6cae70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:37.424011 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-408bf464-7ffe-4a4e-989d-6dfe4a6cae70 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:56:37.425273 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-408bf464-7ffe-4a4e-989d-6dfe4a6cae70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:37.539122 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 650/1313] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:56:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:56:37.815020 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfb19b7-d387-4593-ad7e-b89e8ec76563 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:56:37.816546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-6bfb19b7-d387-4593-ad7e-b89e8ec76563 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:37.917490 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfb19b7-d387-4593-ad7e-b89e8ec76563 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:56:37.988714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:37.991258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ba42e0-8dc7-4e20-8a61-2d1c5b815765 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:56:37.991859 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ba42e0-8dc7-4e20-8a61-2d1c5b815765 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:37.992139 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ba42e0-8dc7-4e20-8a61-2d1c5b815765 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:37.992399 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ba42e0-8dc7-4e20-8a61-2d1c5b815765 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:37.995023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c7ba42e0-8dc7-4e20-8a61-2d1c5b815765 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:38.100289 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ba42e0-8dc7-4e20-8a61-2d1c5b815765 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:56:38.101679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ba42e0-8dc7-4e20-8a61-2d1c5b815765 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:56:38.121134 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 664/1314] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:56:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:56:40.507205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0746c9bf-55db-4449-aedf-3e9d05abcaff None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:40.507705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0746c9bf-55db-4449-aedf-3e9d05abcaff None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:40.507801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0746c9bf-55db-4449-aedf-3e9d05abcaff None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:40.509853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 651/1315] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:56:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:56:40.515013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987dd629-782c-49e0-92e0-250c5926fdb3 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:40.515433 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987dd629-782c-49e0-92e0-250c5926fdb3 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:40.515591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987dd629-782c-49e0-92e0-250c5926fdb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:40.581550 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-987dd629-782c-49e0-92e0-250c5926fdb3 None None] MFA Rules not processed for user `d4a141c6519544a4ad26f923f1fb0382`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:56:40.582752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-987dd629-782c-49e0-92e0-250c5926fdb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:40.637041 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:40.681926 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 665/1316] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:56:40 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:56:40.970847 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0746c9bf-55db-4449-aedf-3e9d05abcaff None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:56:40.972378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-0746c9bf-55db-4449-aedf-3e9d05abcaff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:40.982471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:41.076462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0746c9bf-55db-4449-aedf-3e9d05abcaff None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:56:41.147044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece40d5f-eb11-47c8-a75c-5e7b8b4fa9d7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:56:41.147642 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece40d5f-eb11-47c8-a75c-5e7b8b4fa9d7 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:41.147916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece40d5f-eb11-47c8-a75c-5e7b8b4fa9d7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:41.148185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece40d5f-eb11-47c8-a75c-5e7b8b4fa9d7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:41.150694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ece40d5f-eb11-47c8-a75c-5e7b8b4fa9d7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:41.211147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:41.257721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece40d5f-eb11-47c8-a75c-5e7b8b4fa9d7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:56:41.259188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece40d5f-eb11-47c8-a75c-5e7b8b4fa9d7 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:56:41.277863 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 652/1317] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:56:40 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:56:44.094673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daccb928-75a9-4de0-ad34-ee293d056531 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:44.095213 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daccb928-75a9-4de0-ad34-ee293d056531 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:44.095213 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daccb928-75a9-4de0-ad34-ee293d056531 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:44.096579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 666/1318] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:56:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:56:44.101777 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02439d3b-a171-426e-a11d-f8d64e6b8f02 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:44.102298 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02439d3b-a171-426e-a11d-f8d64e6b8f02 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:44.102298 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02439d3b-a171-426e-a11d-f8d64e6b8f02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:44.166729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-02439d3b-a171-426e-a11d-f8d64e6b8f02 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:56:44.167916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-02439d3b-a171-426e-a11d-f8d64e6b8f02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:44.261893 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 653/1319] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:56:44 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:56:44.545779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daccb928-75a9-4de0-ad34-ee293d056531 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:56:44.547330 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-daccb928-75a9-4de0-ad34-ee293d056531 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:44.650924 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daccb928-75a9-4de0-ad34-ee293d056531 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:56:44.724333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa69efc-7e2e-4884-b726-5786c2320ee5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:56:44.724853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa69efc-7e2e-4884-b726-5786c2320ee5 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:44.725161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa69efc-7e2e-4884-b726-5786c2320ee5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:44.725435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa69efc-7e2e-4884-b726-5786c2320ee5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:44.727563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-afa69efc-7e2e-4884-b726-5786c2320ee5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:44.734962 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:44.846582 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa69efc-7e2e-4884-b726-5786c2320ee5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:56:44.848014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa69efc-7e2e-4884-b726-5786c2320ee5 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:56:44.866017 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 667/1320] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:56:44 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:56:47.824359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2d5a35-984d-4e75-bb1a-90f49a0f380b None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:47.824892 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2d5a35-984d-4e75-bb1a-90f49a0f380b None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:47.824892 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2d5a35-984d-4e75-bb1a-90f49a0f380b None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:47.826304 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 654/1321] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:56:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:56:47.831555 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f716e464-10a6-4758-ade7-8a52015c9bd0 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:47.831962 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f716e464-10a6-4758-ade7-8a52015c9bd0 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:47.832104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f716e464-10a6-4758-ade7-8a52015c9bd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:47.899070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-f716e464-10a6-4758-ade7-8a52015c9bd0 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:56:47.900303 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-f716e464-10a6-4758-ade7-8a52015c9bd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:47.963384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:48.002981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 668/1322] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:56:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:56:48.291712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2d5a35-984d-4e75-bb1a-90f49a0f380b None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:56:48.293363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-3f2d5a35-984d-4e75-bb1a-90f49a0f380b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:48.313572 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:48.403516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2d5a35-984d-4e75-bb1a-90f49a0f380b None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:56:48.480047 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225d6bb8-3053-470d-baa8-5f0788403494 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:56:48.480649 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225d6bb8-3053-470d-baa8-5f0788403494 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:48.480927 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225d6bb8-3053-470d-baa8-5f0788403494 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:48.481229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225d6bb8-3053-470d-baa8-5f0788403494 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:48.482799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-225d6bb8-3053-470d-baa8-5f0788403494 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:48.554687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:48.592444 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225d6bb8-3053-470d-baa8-5f0788403494 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:56:48.593913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225d6bb8-3053-470d-baa8-5f0788403494 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:56:48.611966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 655/1323] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:56:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:56:50.590004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b4cb7b-6c5c-4f49-822e-b57fb2e1a6d5 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:50.590548 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b4cb7b-6c5c-4f49-822e-b57fb2e1a6d5 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:50.590548 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b4cb7b-6c5c-4f49-822e-b57fb2e1a6d5 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:50.592030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 669/1324] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:56:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:56:50.597005 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c2628d-5a32-4ac5-a41b-8d2c90ef45fb None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:50.597482 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c2628d-5a32-4ac5-a41b-8d2c90ef45fb None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:50.597659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c2628d-5a32-4ac5-a41b-8d2c90ef45fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:50.664957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-42c2628d-5a32-4ac5-a41b-8d2c90ef45fb None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:56:50.666285 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-42c2628d-5a32-4ac5-a41b-8d2c90ef45fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:50.763164 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 656/1325] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:56:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:56:51.046333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b4cb7b-6c5c-4f49-822e-b57fb2e1a6d5 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:56:51.048016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c3b4cb7b-6c5c-4f49-822e-b57fb2e1a6d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:51.151695 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b4cb7b-6c5c-4f49-822e-b57fb2e1a6d5 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:56:51.225460 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38a896a-26f8-4925-bf4f-ef91186a2357 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:56:51.225997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38a896a-26f8-4925-bf4f-ef91186a2357 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:51.226284 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38a896a-26f8-4925-bf4f-ef91186a2357 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:51.226555 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38a896a-26f8-4925-bf4f-ef91186a2357 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:51.228638 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e38a896a-26f8-4925-bf4f-ef91186a2357 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:51.248275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:51.347082 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38a896a-26f8-4925-bf4f-ef91186a2357 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:56:51.348608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38a896a-26f8-4925-bf4f-ef91186a2357 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:56:51.368945 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 670/1326] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:56:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:56:53.366117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b58829-409d-4fe5-b67a-85ac1400196f None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:53.366698 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b58829-409d-4fe5-b67a-85ac1400196f None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:53.366698 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b58829-409d-4fe5-b67a-85ac1400196f None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:53.368691 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 657/1327] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:56:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:56:53.374005 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0717b46a-733e-4299-99d7-875981ccdb6d None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:53.374427 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0717b46a-733e-4299-99d7-875981ccdb6d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:53.374557 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0717b46a-733e-4299-99d7-875981ccdb6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:53.444301 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-0717b46a-733e-4299-99d7-875981ccdb6d None None] MFA Rules not processed for user `d4a141c6519544a4ad26f923f1fb0382`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:56:53.445581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0717b46a-733e-4299-99d7-875981ccdb6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:53.518438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:53.551125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 671/1328] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:56:53 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:56:53.840254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b58829-409d-4fe5-b67a-85ac1400196f None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:56:53.841842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-38b58829-409d-4fe5-b67a-85ac1400196f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:53.873388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:53.966906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b58829-409d-4fe5-b67a-85ac1400196f None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:56:54.044431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afa68e3-5d4a-424b-b6da-5ebddde6dab2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:56:54.045034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afa68e3-5d4a-424b-b6da-5ebddde6dab2 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:54.045355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afa68e3-5d4a-424b-b6da-5ebddde6dab2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:54.045631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afa68e3-5d4a-424b-b6da-5ebddde6dab2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:54.047247 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-0afa68e3-5d4a-424b-b6da-5ebddde6dab2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:54.126374 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:54.159071 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afa68e3-5d4a-424b-b6da-5ebddde6dab2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:56:54.160502 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afa68e3-5d4a-424b-b6da-5ebddde6dab2 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:56:54.178255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 658/1329] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:56:53 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:56:56.538149 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e629d9a8-65bf-4d03-8e01-6a9795c3fb15 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:56.538693 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e629d9a8-65bf-4d03-8e01-6a9795c3fb15 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:56.538693 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e629d9a8-65bf-4d03-8e01-6a9795c3fb15 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:56.540180 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 672/1330] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:56:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:56:56.545501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bdc836-adca-4097-8368-39142531e513 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:56.545936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bdc836-adca-4097-8368-39142531e513 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:56.546051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50bdc836-adca-4097-8368-39142531e513 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:56.611385 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-50bdc836-adca-4097-8368-39142531e513 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:56:56.612584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-50bdc836-adca-4097-8368-39142531e513 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:56.706695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 659/1331] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:56:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:56:56.989961 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e629d9a8-65bf-4d03-8e01-6a9795c3fb15 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:56:56.991514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e629d9a8-65bf-4d03-8e01-6a9795c3fb15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:57.096517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e629d9a8-65bf-4d03-8e01-6a9795c3fb15 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:56:57.168730 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8511bf-7de3-4276-b506-14d6614aa65b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:56:57.169357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8511bf-7de3-4276-b506-14d6614aa65b service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:57.169640 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8511bf-7de3-4276-b506-14d6614aa65b service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:57.169918 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8511bf-7de3-4276-b506-14d6614aa65b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:57.171507 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0e8511bf-7de3-4276-b506-14d6614aa65b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:57.198959 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:57.280399 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8511bf-7de3-4276-b506-14d6614aa65b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:56:57.281873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8511bf-7de3-4276-b506-14d6614aa65b service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:56:57.300489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 673/1332] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:56:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:56:59.244657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d33c57b-5d3a-4491-9f10-3e1a376785ba None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:59.245217 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d33c57b-5d3a-4491-9f10-3e1a376785ba None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:59.245217 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d33c57b-5d3a-4491-9f10-3e1a376785ba None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:59.246574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 660/1333] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:56:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:56:59.251793 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a12d48-3430-44b9-b8f1-1356e80a8bb4 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:59.252232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a12d48-3430-44b9-b8f1-1356e80a8bb4 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:59.252354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a12d48-3430-44b9-b8f1-1356e80a8bb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:59.317606 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-e8a12d48-3430-44b9-b8f1-1356e80a8bb4 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:56:59.318821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e8a12d48-3430-44b9-b8f1-1356e80a8bb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:59.393220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:59.416422 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 674/1334] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:56:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:56:59.702441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d33c57b-5d3a-4491-9f10-3e1a376785ba None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:56:59.704002 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6d33c57b-5d3a-4491-9f10-3e1a376785ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:59.744555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:59.806715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d33c57b-5d3a-4491-9f10-3e1a376785ba None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:56:59.878810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547b39d9-0bcb-45cd-a0d3-c6df7a31374a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:56:59.879415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547b39d9-0bcb-45cd-a0d3-c6df7a31374a service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:56:59.879695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547b39d9-0bcb-45cd-a0d3-c6df7a31374a service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:56:59.879968 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547b39d9-0bcb-45cd-a0d3-c6df7a31374a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:56:59.881585 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-547b39d9-0bcb-45cd-a0d3-c6df7a31374a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:56:59.969649 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:56:59.994703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547b39d9-0bcb-45cd-a0d3-c6df7a31374a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:56:59.996416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547b39d9-0bcb-45cd-a0d3-c6df7a31374a service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:57:00.016516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 661/1335] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:56:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:57:02.463290 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93d61d0-40ff-460c-8836-40b0bfcb3fea None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:02.463831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93d61d0-40ff-460c-8836-40b0bfcb3fea None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:02.463831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93d61d0-40ff-460c-8836-40b0bfcb3fea None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:02.465222 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 675/1336] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:57:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:02.470330 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6617cf50-4d65-4e92-a60a-5f5551fca5c9 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:02.470740 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6617cf50-4d65-4e92-a60a-5f5551fca5c9 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:02.470874 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6617cf50-4d65-4e92-a60a-5f5551fca5c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:02.535066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-6617cf50-4d65-4e92-a60a-5f5551fca5c9 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:02.536271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6617cf50-4d65-4e92-a60a-5f5551fca5c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:02.629644 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 662/1337] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:57:02 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:02.930551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93d61d0-40ff-460c-8836-40b0bfcb3fea None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:57:02.931368 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a93d61d0-40ff-460c-8836-40b0bfcb3fea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:03.039446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93d61d0-40ff-460c-8836-40b0bfcb3fea None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:57:03.111882 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e284d8dc-1870-4959-9b0a-f6491c2f284f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:57:03.112490 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e284d8dc-1870-4959-9b0a-f6491c2f284f service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:03.112769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e284d8dc-1870-4959-9b0a-f6491c2f284f service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:03.113042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e284d8dc-1870-4959-9b0a-f6491c2f284f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:03.114673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e284d8dc-1870-4959-9b0a-f6491c2f284f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:03.150692 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:03.222080 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e284d8dc-1870-4959-9b0a-f6491c2f284f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:57:03.223457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e284d8dc-1870-4959-9b0a-f6491c2f284f service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:57:03.249238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 676/1338] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:57:02 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:57:07.436101 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64504fef-c2bc-41e4-bd2d-90761fdabcd1 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:07.436620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64504fef-c2bc-41e4-bd2d-90761fdabcd1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:07.436620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64504fef-c2bc-41e4-bd2d-90761fdabcd1 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:07.438026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 663/1339] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:57:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:07.443173 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396864b8-044a-4b5b-9129-5d5e1ac6ff5a None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:07.443620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396864b8-044a-4b5b-9129-5d5e1ac6ff5a None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:07.443717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396864b8-044a-4b5b-9129-5d5e1ac6ff5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:07.513962 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-396864b8-044a-4b5b-9129-5d5e1ac6ff5a None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:07.515152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-396864b8-044a-4b5b-9129-5d5e1ac6ff5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:07.605961 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:07.617877 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 677/1340] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:57:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:07.925774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b12a4ac-cf82-4f14-88ca-36d459c63bf0 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:07.926998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b12a4ac-cf82-4f14-88ca-36d459c63bf0 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:07.926998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b12a4ac-cf82-4f14-88ca-36d459c63bf0 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:07.927796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 664/1341] 10.222.0.42 () {54 vars in 965 bytes} [Sun Sep 4 00:57:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:07.934087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daedcb98-ac58-4b88-ba95-7876af2580dd None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:07.934371 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daedcb98-ac58-4b88-ba95-7876af2580dd None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:07.934641 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daedcb98-ac58-4b88-ba95-7876af2580dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:08.012906 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-daedcb98-ac58-4b88-ba95-7876af2580dd None None] MFA Rules not processed for user `da4f44b86ca94adf959d9a41d915c902`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:08.014118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-daedcb98-ac58-4b88-ba95-7876af2580dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:08.088953 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 678/1342] 10.222.0.42 () {58 vars in 1062 bytes} [Sun Sep 4 00:57:07 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:08.097838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b12a4ac-cf82-4f14-88ca-36d459c63bf0 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:57:08.099337 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-0b12a4ac-cf82-4f14-88ca-36d459c63bf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:08.147284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:08.208547 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b12a4ac-cf82-4f14-88ca-36d459c63bf0 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:57:08.283042 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf06da9-30b0-4c37-a362-d9e4c036043c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:57:08.283768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf06da9-30b0-4c37-a362-d9e4c036043c service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:08.284044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf06da9-30b0-4c37-a362-d9e4c036043c service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:08.284320 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf06da9-30b0-4c37-a362-d9e4c036043c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:08.286018 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-dcf06da9-30b0-4c37-a362-d9e4c036043c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:08.377351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:08.393735 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf06da9-30b0-4c37-a362-d9e4c036043c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:57:08.395207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf06da9-30b0-4c37-a362-d9e4c036043c service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:57:08.415812 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 665/1343] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:57:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:57:11.931860 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777f11e9-d61a-4af3-8571-c448be867711 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:11.932395 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777f11e9-d61a-4af3-8571-c448be867711 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:11.932395 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777f11e9-d61a-4af3-8571-c448be867711 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:11.935027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 679/1344] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:57:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:11.939048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f916556-cb0e-40d3-92b3-d976bec27c51 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:11.939468 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f916556-cb0e-40d3-92b3-d976bec27c51 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:11.939596 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f916556-cb0e-40d3-92b3-d976bec27c51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:12.009021 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-2f916556-cb0e-40d3-92b3-d976bec27c51 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:12.010302 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2f916556-cb0e-40d3-92b3-d976bec27c51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:12.110199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 666/1345] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:57:11 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:12.395477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777f11e9-d61a-4af3-8571-c448be867711 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:57:12.397124 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-777f11e9-d61a-4af3-8571-c448be867711 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:12.495852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:12.513570 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777f11e9-d61a-4af3-8571-c448be867711 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:57:12.591054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f236ee7b-464b-4216-ba20-84dd09914b8b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:57:12.591666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f236ee7b-464b-4216-ba20-84dd09914b8b service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:12.591942 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f236ee7b-464b-4216-ba20-84dd09914b8b service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:12.592211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f236ee7b-464b-4216-ba20-84dd09914b8b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:12.593886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-f236ee7b-464b-4216-ba20-84dd09914b8b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:12.704489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f236ee7b-464b-4216-ba20-84dd09914b8b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:57:12.705954 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f236ee7b-464b-4216-ba20-84dd09914b8b service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:57:12.724533 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 680/1346] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:57:12 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:57:17.094522 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6671dcc-faac-46a3-825d-e37d722e041f None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:17.095068 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6671dcc-faac-46a3-825d-e37d722e041f None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:17.095068 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6671dcc-faac-46a3-825d-e37d722e041f None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:17.096437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 667/1347] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:57:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:17.101866 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55da972f-8ed4-4bf6-b469-9705d5a23f55 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:17.102364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55da972f-8ed4-4bf6-b469-9705d5a23f55 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:17.102468 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55da972f-8ed4-4bf6-b469-9705d5a23f55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:17.118714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:17.174215 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-55da972f-8ed4-4bf6-b469-9705d5a23f55 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:17.175523 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-55da972f-8ed4-4bf6-b469-9705d5a23f55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:17.277657 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 681/1348] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:57:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:17.561952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6671dcc-faac-46a3-825d-e37d722e041f None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:57:17.563409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a6671dcc-faac-46a3-825d-e37d722e041f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:17.624293 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:17.671281 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6671dcc-faac-46a3-825d-e37d722e041f None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:57:17.746018 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f915b08-0c73-4bf6-9c6f-82ce1788bf6f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:57:17.746623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f915b08-0c73-4bf6-9c6f-82ce1788bf6f service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:17.746902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f915b08-0c73-4bf6-9c6f-82ce1788bf6f service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:17.749891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f915b08-0c73-4bf6-9c6f-82ce1788bf6f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:17.749891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-1f915b08-0c73-4bf6-9c6f-82ce1788bf6f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:17.858295 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:17.861964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f915b08-0c73-4bf6-9c6f-82ce1788bf6f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:57:17.863475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f915b08-0c73-4bf6-9c6f-82ce1788bf6f service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:57:17.883094 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 668/1349] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:57:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:57:22.850566 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb452be-d83c-4991-b2e9-f11e704e794f None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:22.851204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb452be-d83c-4991-b2e9-f11e704e794f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:22.851204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb452be-d83c-4991-b2e9-f11e704e794f None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:22.853137 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 682/1350] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 4 00:57:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:22.860270 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8804afc4-cafd-4182-aa8d-2732fa82a4b7 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:22.860967 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8804afc4-cafd-4182-aa8d-2732fa82a4b7 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:22.860967 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8804afc4-cafd-4182-aa8d-2732fa82a4b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:22.982380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-8804afc4-cafd-4182-aa8d-2732fa82a4b7 None None] MFA Rules not processed for user `b29f444d56ec49308f7fef16f4d1eea5`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:22.983691 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8804afc4-cafd-4182-aa8d-2732fa82a4b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:23.113126 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 669/1351] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 4 00:57:22 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:27.275794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=59 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:27.318990 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e3ba01-e169-4a8b-8ee1-7357bd82748c None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:27.319361 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e3ba01-e169-4a8b-8ee1-7357bd82748c None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:27.319581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e3ba01-e169-4a8b-8ee1-7357bd82748c None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:27.321038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 683/1352] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 4 00:57:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:27.325914 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62addd89-2984-4963-ac79-437aef25c580 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:27.326757 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62addd89-2984-4963-ac79-437aef25c580 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:27.327238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62addd89-2984-4963-ac79-437aef25c580 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:27.387565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:27.413041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-62addd89-2984-4963-ac79-437aef25c580 None None] MFA Rules not processed for user `b29f444d56ec49308f7fef16f4d1eea5`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:27.414828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-62addd89-2984-4963-ac79-437aef25c580 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:27.540051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 670/1353] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 4 00:57:27 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:28.969246 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e7449-40c3-4bff-b288-90061cb30c2e None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:28.969815 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e7449-40c3-4bff-b288-90061cb30c2e None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:28.969975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9e7449-40c3-4bff-b288-90061cb30c2e None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:28.971331 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 684/1354] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 4 00:57:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:28.977379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87367430-e288-4d06-8faf-eb36731cba5e None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:28.977803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87367430-e288-4d06-8faf-eb36731cba5e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:28.977917 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87367430-e288-4d06-8faf-eb36731cba5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:29.055630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-87367430-e288-4d06-8faf-eb36731cba5e None None] MFA Rules not processed for user `b29f444d56ec49308f7fef16f4d1eea5`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:29.056826 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-87367430-e288-4d06-8faf-eb36731cba5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:29.098126 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:29.164230 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 671/1355] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 4 00:57:28 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:32.451107 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cce6ca-6897-45f3-950a-2b891d17e8c1 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:32.451705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cce6ca-6897-45f3-950a-2b891d17e8c1 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:32.451705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cce6ca-6897-45f3-950a-2b891d17e8c1 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:32.452954 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 685/1356] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:57:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:32.461564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761a42f6-c820-466c-8e96-41a3917d4c3c None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:32.462350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761a42f6-c820-466c-8e96-41a3917d4c3c None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:32.462799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761a42f6-c820-466c-8e96-41a3917d4c3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:32.542084 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-761a42f6-c820-466c-8e96-41a3917d4c3c None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:32.546658 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-761a42f6-c820-466c-8e96-41a3917d4c3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:32.677342 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 672/1357] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:57:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:32.979479 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7793123-c596-45d1-a863-e25f9b07742d None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:32.979765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7793123-c596-45d1-a863-e25f9b07742d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:32.980038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7793123-c596-45d1-a863-e25f9b07742d None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:32.981651 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 686/1358] 10.222.0.42 () {54 vars in 962 bytes} [Sun Sep 4 00:57:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:32.992191 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f5091e-f5d9-4dda-8cce-3117cb4fce7e None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:32.992791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f5091e-f5d9-4dda-8cce-3117cb4fce7e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:32.993306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f5091e-f5d9-4dda-8cce-3117cb4fce7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:33.013807 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:33.078081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-75f5091e-f5d9-4dda-8cce-3117cb4fce7e None None] MFA Rules not processed for user `57c88f16c7df40adb917f49804fda21f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:33.079536 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-75f5091e-f5d9-4dda-8cce-3117cb4fce7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:33.206001 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 673/1359] 10.222.0.42 () {58 vars in 1059 bytes} [Sun Sep 4 00:57:32 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 00:57:33.213773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7793123-c596-45d1-a863-e25f9b07742d None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:57:33.215814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d7793123-c596-45d1-a863-e25f9b07742d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:33.377022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7793123-c596-45d1-a863-e25f9b07742d None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:57:33.461893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:33.507260 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0fcd2f-0a9f-498b-9975-3ab68cd9793e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:57:33.508149 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0fcd2f-0a9f-498b-9975-3ab68cd9793e service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:33.508602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0fcd2f-0a9f-498b-9975-3ab68cd9793e service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:33.509029 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0fcd2f-0a9f-498b-9975-3ab68cd9793e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:33.511169 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ef0fcd2f-0a9f-498b-9975-3ab68cd9793e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:33.605567 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0fcd2f-0a9f-498b-9975-3ab68cd9793e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:57:33.609682 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0fcd2f-0a9f-498b-9975-3ab68cd9793e service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:57:33.626248 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 687/1360] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:57:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:57:35.816497 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bc08b5-1b8b-4a64-b505-b0ed237766df None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:35.817055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bc08b5-1b8b-4a64-b505-b0ed237766df None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:35.817055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bc08b5-1b8b-4a64-b505-b0ed237766df None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:35.822853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 674/1361] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:57:35 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:35.824751 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822fc9ee-e73c-491a-af82-61ef72115838 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:35.825252 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822fc9ee-e73c-491a-af82-61ef72115838 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:35.825425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822fc9ee-e73c-491a-af82-61ef72115838 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:35.908137 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-822fc9ee-e73c-491a-af82-61ef72115838 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:35.911612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-822fc9ee-e73c-491a-af82-61ef72115838 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:35.950720 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:36.018322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 688/1362] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:57:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:36.306862 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bc08b5-1b8b-4a64-b505-b0ed237766df None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:57:36.309282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-98bc08b5-1b8b-4a64-b505-b0ed237766df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:36.385360 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:36.470194 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bc08b5-1b8b-4a64-b505-b0ed237766df None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:57:36.595321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a256a671-ff12-4c38-ac8a-6a9470fd5340 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:57:36.595943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a256a671-ff12-4c38-ac8a-6a9470fd5340 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:36.596225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a256a671-ff12-4c38-ac8a-6a9470fd5340 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:36.596497 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a256a671-ff12-4c38-ac8a-6a9470fd5340 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:36.598513 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a256a671-ff12-4c38-ac8a-6a9470fd5340 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:36.624349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:36.699577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a256a671-ff12-4c38-ac8a-6a9470fd5340 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:57:36.701211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a256a671-ff12-4c38-ac8a-6a9470fd5340 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:57:36.719483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 675/1363] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:57:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:57:39.164199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f5d267-0945-4bdd-bdf3-7a5b02b7cc18 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:39.165257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f5d267-0945-4bdd-bdf3-7a5b02b7cc18 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:39.165760 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f5d267-0945-4bdd-bdf3-7a5b02b7cc18 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:39.167330 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 689/1364] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:57:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:39.172636 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ceaef8-9768-4d49-abbe-cf64bd67324a None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:39.173482 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ceaef8-9768-4d49-abbe-cf64bd67324a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:39.173986 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ceaef8-9768-4d49-abbe-cf64bd67324a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:39.243696 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-65ceaef8-9768-4d49-abbe-cf64bd67324a None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:39.245228 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-65ceaef8-9768-4d49-abbe-cf64bd67324a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:39.344173 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:39.347180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 676/1365] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:57:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:39.638769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ae545b-c1d1-43b8-8a4b-af414afcb5bb None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:39.639250 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ae545b-c1d1-43b8-8a4b-af414afcb5bb None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:39.639810 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ae545b-c1d1-43b8-8a4b-af414afcb5bb None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:39.641441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 690/1366] 10.222.0.42 () {54 vars in 962 bytes} [Sun Sep 4 00:57:39 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:39.647626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951c2820-b53c-4076-ae22-5737385979a6 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:39.655859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951c2820-b53c-4076-ae22-5737385979a6 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:39.655859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951c2820-b53c-4076-ae22-5737385979a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:39.730209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-951c2820-b53c-4076-ae22-5737385979a6 None None] MFA Rules not processed for user `57c88f16c7df40adb917f49804fda21f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:39.731427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-951c2820-b53c-4076-ae22-5737385979a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:39.868491 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 677/1367] 10.222.0.42 () {58 vars in 1059 bytes} [Sun Sep 4 00:57:39 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:39.877881 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ae545b-c1d1-43b8-8a4b-af414afcb5bb None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:57:39.879573 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-15ae545b-c1d1-43b8-8a4b-af414afcb5bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:40.038194 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ae545b-c1d1-43b8-8a4b-af414afcb5bb None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:57:40.059731 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:40.166217 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1987657-6505-42a1-a803-2c232baa5496 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:57:40.166885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1987657-6505-42a1-a803-2c232baa5496 service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:40.166982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1987657-6505-42a1-a803-2c232baa5496 service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:40.167254 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1987657-6505-42a1-a803-2c232baa5496 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:40.168876 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d1987657-6505-42a1-a803-2c232baa5496 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:40.261331 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1987657-6505-42a1-a803-2c232baa5496 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:57:40.262943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1987657-6505-42a1-a803-2c232baa5496 service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:57:40.281412 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 691/1368] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:57:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 00:57:40.724759 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec76c06-777b-4e65-8fdf-3c899dd0cfc4 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:40.725319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec76c06-777b-4e65-8fdf-3c899dd0cfc4 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:40.725319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec76c06-777b-4e65-8fdf-3c899dd0cfc4 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:40.726745 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 678/1369] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:57:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:40.731822 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437c3606-7392-4afa-98d6-a9a839c9d283 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:40.732109 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437c3606-7392-4afa-98d6-a9a839c9d283 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:40.732381 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437c3606-7392-4afa-98d6-a9a839c9d283 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:40.765519 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:40.805938 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-437c3606-7392-4afa-98d6-a9a839c9d283 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:40.807528 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-437c3606-7392-4afa-98d6-a9a839c9d283 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:40.914553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 692/1370] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:57:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:41.516804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6216529d-d64b-42fa-b52e-af152de566f0 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:41.516804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6216529d-d64b-42fa-b52e-af152de566f0 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:41.516804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6216529d-d64b-42fa-b52e-af152de566f0 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:41.517771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 679/1371] 10.222.0.42 () {54 vars in 964 bytes} [Sun Sep 4 00:57:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:41.523894 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d5bd8b-f5b5-4bfb-9239-f59d857ff554 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:41.524357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d5bd8b-f5b5-4bfb-9239-f59d857ff554 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:41.524458 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d5bd8b-f5b5-4bfb-9239-f59d857ff554 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:41.608304 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-b9d5bd8b-f5b5-4bfb-9239-f59d857ff554 None None] MFA Rules not processed for user `615a607b6be14b82979f94eabaf4c34c`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:41.609568 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b9d5bd8b-f5b5-4bfb-9239-f59d857ff554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:41.661606 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:41.691546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 693/1372] 10.222.0.42 () {58 vars in 1061 bytes} [Sun Sep 4 00:57:41 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 00:57:41.699899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6216529d-d64b-42fa-b52e-af152de566f0 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:57:41.701507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6216529d-d64b-42fa-b52e-af152de566f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:41.756857 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:41.814766 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6216529d-d64b-42fa-b52e-af152de566f0 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:57:41.899965 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9e7765-ba9b-4153-9276-d6ede7de9329 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:57:41.900579 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9e7765-ba9b-4153-9276-d6ede7de9329 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:41.900859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9e7765-ba9b-4153-9276-d6ede7de9329 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:41.901163 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9e7765-ba9b-4153-9276-d6ede7de9329 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:41.902940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-bd9e7765-ba9b-4153-9276-d6ede7de9329 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:42.003844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:42.017533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9e7765-ba9b-4153-9276-d6ede7de9329 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:57:42.018863 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9e7765-ba9b-4153-9276-d6ede7de9329 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:57:42.039168 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 680/1373] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:57:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:57:42.456440 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e2d3ab-b8b4-4aba-8787-3d55a915203f None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:42.456729 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e2d3ab-b8b4-4aba-8787-3d55a915203f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:42.457002 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e2d3ab-b8b4-4aba-8787-3d55a915203f None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:42.459219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 694/1374] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:57:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:42.464610 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8b4069-d868-4256-8056-37f612788e6e None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:42.464903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8b4069-d868-4256-8056-37f612788e6e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:42.465225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8b4069-d868-4256-8056-37f612788e6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:42.532457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-3b8b4069-d868-4256-8056-37f612788e6e None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:42.533622 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-3b8b4069-d868-4256-8056-37f612788e6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:42.633145 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 681/1375] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:57:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:42.926188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e2d3ab-b8b4-4aba-8787-3d55a915203f None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:57:42.928320 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d4e2d3ab-b8b4-4aba-8787-3d55a915203f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:43.075234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e2d3ab-b8b4-4aba-8787-3d55a915203f None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:57:43.126085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:43.189207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16be588a-3d5b-4f0b-88b3-de0c866054a7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:57:43.189847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16be588a-3d5b-4f0b-88b3-de0c866054a7 service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:43.190127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16be588a-3d5b-4f0b-88b3-de0c866054a7 service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:43.190396 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16be588a-3d5b-4f0b-88b3-de0c866054a7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:43.192088 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-16be588a-3d5b-4f0b-88b3-de0c866054a7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:43.281802 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16be588a-3d5b-4f0b-88b3-de0c866054a7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:57:43.283412 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16be588a-3d5b-4f0b-88b3-de0c866054a7 service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:57:43.302921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 695/1376] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:57:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:57:45.434679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc733bf-bbb0-4cc3-991d-0cdfb11993fb None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:45.435207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc733bf-bbb0-4cc3-991d-0cdfb11993fb None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:45.435207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc733bf-bbb0-4cc3-991d-0cdfb11993fb None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:45.436715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 682/1377] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:57:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:45.441699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136d9e6a-d485-41c1-9ac2-a0e852cfcfa8 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:45.442593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136d9e6a-d485-41c1-9ac2-a0e852cfcfa8 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:45.442818 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136d9e6a-d485-41c1-9ac2-a0e852cfcfa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:45.513523 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-136d9e6a-d485-41c1-9ac2-a0e852cfcfa8 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:45.514721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-136d9e6a-d485-41c1-9ac2-a0e852cfcfa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:45.521981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:45.641067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 696/1378] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:57:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:45.954583 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc733bf-bbb0-4cc3-991d-0cdfb11993fb None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:57:45.956350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-3dc733bf-bbb0-4cc3-991d-0cdfb11993fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:46.027090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:46.124058 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc733bf-bbb0-4cc3-991d-0cdfb11993fb None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:57:46.251470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c95171-f434-47e7-a0e9-bc266dd752c6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:57:46.252241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c95171-f434-47e7-a0e9-bc266dd752c6 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:46.252629 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c95171-f434-47e7-a0e9-bc266dd752c6 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:46.253004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c95171-f434-47e7-a0e9-bc266dd752c6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:46.255276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a3c95171-f434-47e7-a0e9-bc266dd752c6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:46.277221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:46.355745 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c95171-f434-47e7-a0e9-bc266dd752c6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:57:46.357527 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c95171-f434-47e7-a0e9-bc266dd752c6 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:57:46.377434 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 683/1379] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:57:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:57:48.494449 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2023b38-4f3b-4c78-acbc-adda4cf20256 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:48.495002 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2023b38-4f3b-4c78-acbc-adda4cf20256 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:48.495002 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2023b38-4f3b-4c78-acbc-adda4cf20256 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:48.496572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 697/1380] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:57:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:48.502585 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d02499d-ec69-42e5-8582-b7044afe58a1 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:48.503143 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d02499d-ec69-42e5-8582-b7044afe58a1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:48.503406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d02499d-ec69-42e5-8582-b7044afe58a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:48.580851 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-3d02499d-ec69-42e5-8582-b7044afe58a1 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:48.582089 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-3d02499d-ec69-42e5-8582-b7044afe58a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:48.818014 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:48.829265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 684/1381] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:57:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 328 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:49.133051 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2023b38-4f3b-4c78-acbc-adda4cf20256 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:57:49.134572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b2023b38-4f3b-4c78-acbc-adda4cf20256 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:49.282099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:49.292887 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2023b38-4f3b-4c78-acbc-adda4cf20256 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:57:49.415643 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143db8d6-5d87-4ddd-a6de-3e56e726fff1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:57:49.416157 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143db8d6-5d87-4ddd-a6de-3e56e726fff1 service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:49.416431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143db8d6-5d87-4ddd-a6de-3e56e726fff1 service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:49.416699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143db8d6-5d87-4ddd-a6de-3e56e726fff1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:49.418416 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-143db8d6-5d87-4ddd-a6de-3e56e726fff1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:49.511983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143db8d6-5d87-4ddd-a6de-3e56e726fff1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:57:49.513612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143db8d6-5d87-4ddd-a6de-3e56e726fff1 service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:57:49.523489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:49.531760 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 698/1382] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:57:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:57:51.818449 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db39138-e0cd-40cb-bf39-89253f8746b9 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:51.818999 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db39138-e0cd-40cb-bf39-89253f8746b9 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:51.818999 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db39138-e0cd-40cb-bf39-89253f8746b9 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:51.820335 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 685/1383] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:57:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:51.825748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925ca2d0-bdfd-475c-80b3-165cb0d3fdbc None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:51.826342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925ca2d0-bdfd-475c-80b3-165cb0d3fdbc None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:51.826342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925ca2d0-bdfd-475c-80b3-165cb0d3fdbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:51.903189 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-925ca2d0-bdfd-475c-80b3-165cb0d3fdbc None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:51.904778 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-925ca2d0-bdfd-475c-80b3-165cb0d3fdbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:52.025353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 699/1384] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:57:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:52.312077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db39138-e0cd-40cb-bf39-89253f8746b9 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:57:52.313612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-0db39138-e0cd-40cb-bf39-89253f8746b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:52.466831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db39138-e0cd-40cb-bf39-89253f8746b9 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:57:52.545325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:52.593062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fbee38-0a39-4e25-9ebd-f8beccff4f47 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:57:52.593687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fbee38-0a39-4e25-9ebd-f8beccff4f47 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:52.593965 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fbee38-0a39-4e25-9ebd-f8beccff4f47 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:52.594232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fbee38-0a39-4e25-9ebd-f8beccff4f47 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:52.595868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-95fbee38-0a39-4e25-9ebd-f8beccff4f47 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:52.699055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fbee38-0a39-4e25-9ebd-f8beccff4f47 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:57:52.701194 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fbee38-0a39-4e25-9ebd-f8beccff4f47 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:57:52.724898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 686/1385] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:57:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:57:54.732700 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0f8273-785c-4f90-a353-ed141fa8294f None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:54.733460 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0f8273-785c-4f90-a353-ed141fa8294f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:54.733460 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0f8273-785c-4f90-a353-ed141fa8294f None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:54.735195 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 700/1386] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:57:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:54.742470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e755d52d-a92c-48c7-b98c-18c21a60ceb8 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:54.742960 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e755d52d-a92c-48c7-b98c-18c21a60ceb8 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:54.743170 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e755d52d-a92c-48c7-b98c-18c21a60ceb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:54.812896 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-e755d52d-a92c-48c7-b98c-18c21a60ceb8 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:54.814140 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e755d52d-a92c-48c7-b98c-18c21a60ceb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:54.851683 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:54.932769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 687/1387] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:57:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:55.231111 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0f8273-785c-4f90-a353-ed141fa8294f None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:57:55.232636 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7a0f8273-785c-4f90-a353-ed141fa8294f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:55.311376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:55.389488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0f8273-785c-4f90-a353-ed141fa8294f None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:57:55.510284 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca85e5d-a4eb-4c73-8182-3a8efbde8748 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:57:55.510822 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca85e5d-a4eb-4c73-8182-3a8efbde8748 service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:55.511103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca85e5d-a4eb-4c73-8182-3a8efbde8748 service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:55.511379 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca85e5d-a4eb-4c73-8182-3a8efbde8748 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:55.512980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-1ca85e5d-a4eb-4c73-8182-3a8efbde8748 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:55.554421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:55.606530 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca85e5d-a4eb-4c73-8182-3a8efbde8748 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:57:55.608142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca85e5d-a4eb-4c73-8182-3a8efbde8748 service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:57:55.627720 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 701/1388] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:57:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:57:57.405357 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ece6c58-bb5f-4873-91d3-612510cd0e81 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:57.405904 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ece6c58-bb5f-4873-91d3-612510cd0e81 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:57.405904 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ece6c58-bb5f-4873-91d3-612510cd0e81 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:57.407232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 688/1389] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:57:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:57.413011 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75299b54-61f2-4f53-97e1-410822665566 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:57.413483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75299b54-61f2-4f53-97e1-410822665566 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:57.413606 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75299b54-61f2-4f53-97e1-410822665566 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:57.483748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-75299b54-61f2-4f53-97e1-410822665566 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:57.484958 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-75299b54-61f2-4f53-97e1-410822665566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:57.583087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 702/1390] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:57:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:57.764822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dead45-cf56-4368-9a44-a874e093fc29 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:57.765160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dead45-cf56-4368-9a44-a874e093fc29 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:57.765273 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dead45-cf56-4368-9a44-a874e093fc29 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:57.766851 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 689/1391] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:57:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:57.773435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc458b9b-e5a0-4750-8516-8c72c6853e07 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:57.773754 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc458b9b-e5a0-4750-8516-8c72c6853e07 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:57.774113 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc458b9b-e5a0-4750-8516-8c72c6853e07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:57.791444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:57.846910 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-fc458b9b-e5a0-4750-8516-8c72c6853e07 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:57.849187 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-fc458b9b-e5a0-4750-8516-8c72c6853e07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:57.899206 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fb0062-83a2-4042-9840-d96452ca851d None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:57.901858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fb0062-83a2-4042-9840-d96452ca851d None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:57.901858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fb0062-83a2-4042-9840-d96452ca851d None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:57.902185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 690/1392] 10.222.0.42 () {54 vars in 961 bytes} [Sun Sep 4 00:57:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:57:57.907347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2898ca34-5049-4117-b364-28f5b44ae340 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:57.907765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2898ca34-5049-4117-b364-28f5b44ae340 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:57.908112 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2898ca34-5049-4117-b364-28f5b44ae340 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:57.961573 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 703/1393] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:57:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:57.986616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-2898ca34-5049-4117-b364-28f5b44ae340 None None] MFA Rules not processed for user `69153344fc4b45588bdbcecdfcfd7c3f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:57.987809 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2898ca34-5049-4117-b364-28f5b44ae340 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:58.063855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 691/1394] 10.222.0.42 () {58 vars in 1058 bytes} [Sun Sep 4 00:57:57 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 00:57:58.072066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc458b9b-e5a0-4750-8516-8c72c6853e07 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:57:58.073410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-fc458b9b-e5a0-4750-8516-8c72c6853e07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:58.164407 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc458b9b-e5a0-4750-8516-8c72c6853e07 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:57:58.172033 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:58.238886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab78a05c-d7d8-4878-9b0f-6dbad78046b9 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '69153344fc4b45588bdbcecdfcfd7c3f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:57:58.239489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab78a05c-d7d8-4878-9b0f-6dbad78046b9 service glance] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:58.239758 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab78a05c-d7d8-4878-9b0f-6dbad78046b9 service glance] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:58.240022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab78a05c-d7d8-4878-9b0f-6dbad78046b9 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:58.241887 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ab78a05c-d7d8-4878-9b0f-6dbad78046b9 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:58.260576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2898ca34-5049-4117-b364-28f5b44ae340 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:57:58.262258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2898ca34-5049-4117-b364-28f5b44ae340 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:58.292202 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:58.353933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab78a05c-d7d8-4878-9b0f-6dbad78046b9 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:57:58.355286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab78a05c-d7d8-4878-9b0f-6dbad78046b9 service glance] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:57:58.374456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 704/1395] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:57:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:57:58.381440 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2898ca34-5049-4117-b364-28f5b44ae340 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:57:58.495153 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591b0811-5305-4215-a44b-57fb6b7005db service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:57:58.495809 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591b0811-5305-4215-a44b-57fb6b7005db service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:58.496037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591b0811-5305-4215-a44b-57fb6b7005db service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:58.496313 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591b0811-5305-4215-a44b-57fb6b7005db service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:58.497960 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-591b0811-5305-4215-a44b-57fb6b7005db service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:58.517349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:58.592744 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591b0811-5305-4215-a44b-57fb6b7005db service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:57:58.594424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591b0811-5305-4215-a44b-57fb6b7005db service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:57:58.615757 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 692/1396] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:57:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 00:57:59.275449 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4481aed5-f96c-4419-a892-f506274260fe None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:59.275948 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4481aed5-f96c-4419-a892-f506274260fe None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:59.276125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4481aed5-f96c-4419-a892-f506274260fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:59.322933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:59.344620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-4481aed5-f96c-4419-a892-f506274260fe None None] MFA Rules not processed for user `70cef7c271e046a3b49230057cc2d28f`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:59.345863 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4481aed5-f96c-4419-a892-f506274260fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:59.434494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 705/1397] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 4 00:57:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:57:59.480769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a5648b-66d0-4d5b-8ab5-94b0b98ea5f1 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:59.481169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a5648b-66d0-4d5b-8ab5-94b0b98ea5f1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:59.481511 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a5648b-66d0-4d5b-8ab5-94b0b98ea5f1 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:59.482923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 693/1398] 10.222.0.42 () {54 vars in 943 bytes} [Sun Sep 4 00:57:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:57:59.489938 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e0a091-f194-4a4a-8ef4-a4df3d1da052 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:59.490229 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e0a091-f194-4a4a-8ef4-a4df3d1da052 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:59.490502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e0a091-f194-4a4a-8ef4-a4df3d1da052 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:59.567820 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-f7e0a091-f194-4a4a-8ef4-a4df3d1da052 None None] MFA Rules not processed for user `123ab6ff796c4e599ce1d6f4095bd898`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:57:59.569026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-f7e0a091-f194-4a4a-8ef4-a4df3d1da052 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:59.632505 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:59.678348 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 706/1399] 10.222.0.42 () {58 vars in 1040 bytes} [Sun Sep 4 00:57:59 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 00:57:59.687515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a5648b-66d0-4d5b-8ab5-94b0b98ea5f1 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:57:59.689004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-63a5648b-66d0-4d5b-8ab5-94b0b98ea5f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:57:59.825916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a5648b-66d0-4d5b-8ab5-94b0b98ea5f1 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:57:59.835861 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:57:59.933207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b3c941-24b7-4848-95f3-46dbd290586a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '123ab6ff796c4e599ce1d6f4095bd898', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:57:59.933873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b3c941-24b7-4848-95f3-46dbd290586a service swift] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:57:59.934146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b3c941-24b7-4848-95f3-46dbd290586a service swift] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:57:59.934410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b3c941-24b7-4848-95f3-46dbd290586a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:57:59.936027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a3b3c941-24b7-4848-95f3-46dbd290586a service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:00.019581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b3c941-24b7-4848-95f3-46dbd290586a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:00.021180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b3c941-24b7-4848-95f3-46dbd290586a service swift] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:00.028961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 694/1400] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 4 00:57:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 04 00:58:00.791857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f431eab2-7a58-49ce-b558-3cd36160b7d7 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:00.792391 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f431eab2-7a58-49ce-b558-3cd36160b7d7 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:00.792681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f431eab2-7a58-49ce-b558-3cd36160b7d7 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:00.794298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 707/1401] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:58:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:00.799400 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94634855-4d5d-4ccb-a3f8-d38521944faa None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:00.799839 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94634855-4d5d-4ccb-a3f8-d38521944faa None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:00.800099 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94634855-4d5d-4ccb-a3f8-d38521944faa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:00.853831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:00.875976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-94634855-4d5d-4ccb-a3f8-d38521944faa None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:00.877380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-94634855-4d5d-4ccb-a3f8-d38521944faa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:00.989406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 695/1402] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:58:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:01.288765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f431eab2-7a58-49ce-b558-3cd36160b7d7 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:01.290328 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-f431eab2-7a58-49ce-b558-3cd36160b7d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:01.466473 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f431eab2-7a58-49ce-b558-3cd36160b7d7 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:01.513670 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:01.607396 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983e2f9f-b0f8-48dd-ac54-c61aebeb4a8c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:01.608009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983e2f9f-b0f8-48dd-ac54-c61aebeb4a8c service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:01.608295 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983e2f9f-b0f8-48dd-ac54-c61aebeb4a8c service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:01.608564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983e2f9f-b0f8-48dd-ac54-c61aebeb4a8c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:01.610203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-983e2f9f-b0f8-48dd-ac54-c61aebeb4a8c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:01.719000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983e2f9f-b0f8-48dd-ac54-c61aebeb4a8c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:01.720605 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983e2f9f-b0f8-48dd-ac54-c61aebeb4a8c service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:01.741097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 708/1403] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:58:03.890185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdda0a7-3397-4eb1-8959-ddce4427683a None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:03.890834 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdda0a7-3397-4eb1-8959-ddce4427683a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:03.890834 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdda0a7-3397-4eb1-8959-ddce4427683a None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:03.892266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 696/1404] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:58:03 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:03.900804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db8c048-6124-4b08-861b-f70eaa90dcb5 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:03.902934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db8c048-6124-4b08-861b-f70eaa90dcb5 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:03.902934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db8c048-6124-4b08-861b-f70eaa90dcb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:03.976652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:03.989243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-3db8c048-6124-4b08-861b-f70eaa90dcb5 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:03.990487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-3db8c048-6124-4b08-861b-f70eaa90dcb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:04.123809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 709/1405] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:58:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:04.477782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdda0a7-3397-4eb1-8959-ddce4427683a None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:04.479899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-bcdda0a7-3397-4eb1-8959-ddce4427683a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:04.637184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:04.685375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdda0a7-3397-4eb1-8959-ddce4427683a None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:04.837569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45173392-be0c-4766-892e-4fea431f1386 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:04.838196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45173392-be0c-4766-892e-4fea431f1386 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:04.838475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45173392-be0c-4766-892e-4fea431f1386 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:04.838744 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45173392-be0c-4766-892e-4fea431f1386 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:04.840606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-45173392-be0c-4766-892e-4fea431f1386 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:04.933208 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:04.945295 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45173392-be0c-4766-892e-4fea431f1386 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:04.948475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45173392-be0c-4766-892e-4fea431f1386 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:04.969521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 697/1406] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 493 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:58:07.160196 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c084e90-ed36-4dab-a10c-d26ad40b41d7 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:07.160777 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c084e90-ed36-4dab-a10c-d26ad40b41d7 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:07.161040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c084e90-ed36-4dab-a10c-d26ad40b41d7 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:07.164582 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 710/1407] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:58:07 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:07.172151 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac3a56a-066d-4284-a327-a5a7fda006ac None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:07.172719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac3a56a-066d-4284-a327-a5a7fda006ac None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:07.172719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac3a56a-066d-4284-a327-a5a7fda006ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:07.251879 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-6ac3a56a-066d-4284-a327-a5a7fda006ac None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:07.253799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6ac3a56a-066d-4284-a327-a5a7fda006ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:07.401146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 698/1408] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:58:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:07.672826 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c084e90-ed36-4dab-a10c-d26ad40b41d7 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:07.674397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0c084e90-ed36-4dab-a10c-d26ad40b41d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:07.826427 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:07.862645 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c084e90-ed36-4dab-a10c-d26ad40b41d7 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:08.008152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faae123-8280-45f4-abe5-f0f07eec572f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:08.008767 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faae123-8280-45f4-abe5-f0f07eec572f service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:08.009140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faae123-8280-45f4-abe5-f0f07eec572f service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:08.009442 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faae123-8280-45f4-abe5-f0f07eec572f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:08.011085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-8faae123-8280-45f4-abe5-f0f07eec572f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:08.126704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8faae123-8280-45f4-abe5-f0f07eec572f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:08.128294 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8faae123-8280-45f4-abe5-f0f07eec572f service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:08.153882 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 711/1409] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 482 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:58:08.257563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:17.369008 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=74 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:18.513819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=7 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:23.914243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fe4485-10ac-4ee5-a989-644c2e0bc3ad None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:23.914816 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fe4485-10ac-4ee5-a989-644c2e0bc3ad None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:23.914816 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fe4485-10ac-4ee5-a989-644c2e0bc3ad None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:23.916148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 699/1410] 10.222.0.42 () {54 vars in 918 bytes} [Sun Sep 4 00:58:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:23.921404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2561e9bc-9156-4ee4-b298-5bdd1897415d None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:23.921841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2561e9bc-9156-4ee4-b298-5bdd1897415d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:23.921944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2561e9bc-9156-4ee4-b298-5bdd1897415d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:23.998592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-2561e9bc-9156-4ee4-b298-5bdd1897415d None None] MFA Rules not processed for user `b29f444d56ec49308f7fef16f4d1eea5`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:23.999829 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-2561e9bc-9156-4ee4-b298-5bdd1897415d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:24.105413 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 712/1411] 10.222.0.42 () {58 vars in 1015 bytes} [Sun Sep 4 00:58:23 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:24.120715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a950f4a-8067-4e0e-92a2-ebd5fcbd32d7 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:24.121000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a950f4a-8067-4e0e-92a2-ebd5fcbd32d7 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:24.121306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a950f4a-8067-4e0e-92a2-ebd5fcbd32d7 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:24.122643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 700/1412] 10.222.0.42 () {54 vars in 918 bytes} [Sun Sep 4 00:58:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:24.127077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbcf9e3-ce8e-4a65-83d5-0056efaeef7d None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:24.127360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbcf9e3-ce8e-4a65-83d5-0056efaeef7d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:24.127633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbcf9e3-ce8e-4a65-83d5-0056efaeef7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:24.198800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:24.201791 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-0cbcf9e3-ce8e-4a65-83d5-0056efaeef7d None None] MFA Rules not processed for user `57c88f16c7df40adb917f49804fda21f`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:24.203141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0cbcf9e3-ce8e-4a65-83d5-0056efaeef7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:24.319770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 713/1413] 10.222.0.42 () {58 vars in 1015 bytes} [Sun Sep 4 00:58:24 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:26.735900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f22c541-b6b5-4aa6-ab14-13f0da8eb536 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:26.736415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f22c541-b6b5-4aa6-ab14-13f0da8eb536 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:26.736415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f22c541-b6b5-4aa6-ab14-13f0da8eb536 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:26.737696 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 701/1414] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 4 00:58:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:26.743740 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac34565-2899-4e70-a3af-d1fb88045706 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:26.744141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac34565-2899-4e70-a3af-d1fb88045706 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:26.744292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac34565-2899-4e70-a3af-d1fb88045706 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:26.809807 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-7ac34565-2899-4e70-a3af-d1fb88045706 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:26.811047 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7ac34565-2899-4e70-a3af-d1fb88045706 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:26.865714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:26.906408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 714/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 4 00:58:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:27.202160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f22c541-b6b5-4aa6-ab14-13f0da8eb536 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:27.204360 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8f22c541-b6b5-4aa6-ab14-13f0da8eb536 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:27.351905 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f22c541-b6b5-4aa6-ab14-13f0da8eb536 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:27.428456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:27.469129 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9171cb-c356-49df-9169-218fd3463ba4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:27.469733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9171cb-c356-49df-9169-218fd3463ba4 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:27.470009 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9171cb-c356-49df-9169-218fd3463ba4 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:27.470272 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9171cb-c356-49df-9169-218fd3463ba4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:27.471837 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6f9171cb-c356-49df-9169-218fd3463ba4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:27.562357 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9171cb-c356-49df-9169-218fd3463ba4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:27.563952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9171cb-c356-49df-9169-218fd3463ba4 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:27.582092 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 702/1416] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:58:28.493593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47a8a63-f4bf-4d09-bce4-e82354ed29db None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:28.494135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47a8a63-f4bf-4d09-bce4-e82354ed29db None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:28.494235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47a8a63-f4bf-4d09-bce4-e82354ed29db None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:28.495524 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 715/1417] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 4 00:58:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:28.500347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64e4c5b-ddec-4501-84f9-19296945677e None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:28.500777 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64e4c5b-ddec-4501-84f9-19296945677e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:28.500904 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64e4c5b-ddec-4501-84f9-19296945677e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:28.576590 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-c64e4c5b-ddec-4501-84f9-19296945677e None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:28.577857 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-c64e4c5b-ddec-4501-84f9-19296945677e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:28.605220 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:28.675160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 703/1418] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 4 00:58:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:28.828567 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47a8a63-f4bf-4d09-bce4-e82354ed29db None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:28.830196 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e47a8a63-f4bf-4d09-bce4-e82354ed29db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:28.978813 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47a8a63-f4bf-4d09-bce4-e82354ed29db None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:29.020649 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:29.094644 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9c8a08-faa6-43af-a55a-b4b0b4b3f169 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:29.095173 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9c8a08-faa6-43af-a55a-b4b0b4b3f169 service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:29.095453 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9c8a08-faa6-43af-a55a-b4b0b4b3f169 service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:29.095726 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9c8a08-faa6-43af-a55a-b4b0b4b3f169 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:29.097356 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-be9c8a08-faa6-43af-a55a-b4b0b4b3f169 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:29.188413 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9c8a08-faa6-43af-a55a-b4b0b4b3f169 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:29.190035 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9c8a08-faa6-43af-a55a-b4b0b4b3f169 service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:29.209311 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 716/1419] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:58:30.084908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b68d72f-cb4a-4bf8-ab38-b88397ca4547 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:30.085519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b68d72f-cb4a-4bf8-ab38-b88397ca4547 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:30.085519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b68d72f-cb4a-4bf8-ab38-b88397ca4547 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:30.086833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 704/1420] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 4 00:58:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:30.091633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94cd440-63c7-4cd8-b20d-09e7ddeebfdb None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:30.092058 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94cd440-63c7-4cd8-b20d-09e7ddeebfdb None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:30.092213 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94cd440-63c7-4cd8-b20d-09e7ddeebfdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:30.146330 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:30.167205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-b94cd440-63c7-4cd8-b20d-09e7ddeebfdb None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:30.168406 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b94cd440-63c7-4cd8-b20d-09e7ddeebfdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:30.262574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 717/1421] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 4 00:58:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:30.414269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b68d72f-cb4a-4bf8-ab38-b88397ca4547 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:30.415737 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-1b68d72f-cb4a-4bf8-ab38-b88397ca4547 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:30.562832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b68d72f-cb4a-4bf8-ab38-b88397ca4547 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:30.575764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:30.679274 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd285a72-9c57-486e-99e3-a170824b21ee service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:30.679814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd285a72-9c57-486e-99e3-a170824b21ee service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:30.680097 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd285a72-9c57-486e-99e3-a170824b21ee service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:30.680369 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd285a72-9c57-486e-99e3-a170824b21ee service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:30.682040 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-dd285a72-9c57-486e-99e3-a170824b21ee service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:30.771196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd285a72-9c57-486e-99e3-a170824b21ee service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:30.772781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd285a72-9c57-486e-99e3-a170824b21ee service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:30.790372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 705/1422] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:58:31.652463 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23366e05-c23c-419c-a4de-19f6813be2c2 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:31.653031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23366e05-c23c-419c-a4de-19f6813be2c2 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:31.653031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23366e05-c23c-419c-a4de-19f6813be2c2 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:31.654390 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 718/1423] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 4 00:58:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:31.659416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fb91d9-ee90-4740-b708-355fcb5074e5 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:31.659794 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fb91d9-ee90-4740-b708-355fcb5074e5 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:31.659965 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fb91d9-ee90-4740-b708-355fcb5074e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:31.681657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:31.736943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-a7fb91d9-ee90-4740-b708-355fcb5074e5 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:31.738096 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a7fb91d9-ee90-4740-b708-355fcb5074e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:31.833762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 706/1424] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 4 00:58:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:31.991392 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23366e05-c23c-419c-a4de-19f6813be2c2 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:31.992945 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-23366e05-c23c-419c-a4de-19f6813be2c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:32.102401 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:32.138428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23366e05-c23c-419c-a4de-19f6813be2c2 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:32.251162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d75bef6-fdca-43df-8a3b-d807c06d3e2d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:32.251703 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d75bef6-fdca-43df-8a3b-d807c06d3e2d service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:32.251975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d75bef6-fdca-43df-8a3b-d807c06d3e2d service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:32.252240 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d75bef6-fdca-43df-8a3b-d807c06d3e2d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:32.253843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-3d75bef6-fdca-43df-8a3b-d807c06d3e2d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:32.333457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:32.346536 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d75bef6-fdca-43df-8a3b-d807c06d3e2d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:32.348229 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d75bef6-fdca-43df-8a3b-d807c06d3e2d service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:32.365974 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 719/1425] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:58:33.233456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636c4934-e10f-4067-b1c3-464d683db947 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:33.233970 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636c4934-e10f-4067-b1c3-464d683db947 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:33.233970 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636c4934-e10f-4067-b1c3-464d683db947 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:33.235296 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 707/1426] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 4 00:58:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:33.241331 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e28acae-bd0b-4ec6-98ce-eda65d110129 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:33.241737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e28acae-bd0b-4ec6-98ce-eda65d110129 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:33.241887 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e28acae-bd0b-4ec6-98ce-eda65d110129 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:33.330289 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-5e28acae-bd0b-4ec6-98ce-eda65d110129 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:33.331567 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-5e28acae-bd0b-4ec6-98ce-eda65d110129 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:33.433238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 720/1427] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 4 00:58:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:33.584454 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636c4934-e10f-4067-b1c3-464d683db947 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:33.585994 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-636c4934-e10f-4067-b1c3-464d683db947 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:33.670314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:33.731428 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636c4934-e10f-4067-b1c3-464d683db947 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:33.894379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbafd1d-6f8c-43ed-a513-48eee1a46432 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:33.894379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbafd1d-6f8c-43ed-a513-48eee1a46432 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:33.894379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbafd1d-6f8c-43ed-a513-48eee1a46432 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:33.894379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbafd1d-6f8c-43ed-a513-48eee1a46432 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:33.895669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6dbafd1d-6f8c-43ed-a513-48eee1a46432 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:33.895669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:33.935283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbafd1d-6f8c-43ed-a513-48eee1a46432 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:33.936863 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbafd1d-6f8c-43ed-a513-48eee1a46432 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:33.955089 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 708/1428] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 00:58:34.802960 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f35322-7e72-408b-96d1-a99ce3de3593 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:34.803511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f35322-7e72-408b-96d1-a99ce3de3593 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:34.803511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f35322-7e72-408b-96d1-a99ce3de3593 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:34.804786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 721/1429] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 4 00:58:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:34.809757 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b78e06f-2dcb-460e-8719-9951bb8f6ecf None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:34.810213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b78e06f-2dcb-460e-8719-9951bb8f6ecf None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:34.810325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b78e06f-2dcb-460e-8719-9951bb8f6ecf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:34.886868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-7b78e06f-2dcb-460e-8719-9951bb8f6ecf None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:34.888094 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-7b78e06f-2dcb-460e-8719-9951bb8f6ecf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:34.984068 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 709/1430] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 4 00:58:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:35.135995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f35322-7e72-408b-96d1-a99ce3de3593 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:35.137541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-24f35322-7e72-408b-96d1-a99ce3de3593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:35.185551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:35.280963 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f35322-7e72-408b-96d1-a99ce3de3593 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:35.392308 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7387773d-a59c-40de-8ea4-2350010689be service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:35.392830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7387773d-a59c-40de-8ea4-2350010689be service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:35.393126 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7387773d-a59c-40de-8ea4-2350010689be service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:35.393397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7387773d-a59c-40de-8ea4-2350010689be service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:35.394983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7387773d-a59c-40de-8ea4-2350010689be service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:35.401734 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:35.486456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7387773d-a59c-40de-8ea4-2350010689be service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:35.488185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7387773d-a59c-40de-8ea4-2350010689be service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:35.506911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 722/1431] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:58:36.401981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea56bbb2-354b-4b70-8c37-652b1a1df3f1 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:36.402561 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea56bbb2-354b-4b70-8c37-652b1a1df3f1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:36.402561 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea56bbb2-354b-4b70-8c37-652b1a1df3f1 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:36.403853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 710/1432] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 4 00:58:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:36.408841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9a02d6-82e5-4f45-8b75-e94d9f0b17c5 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:36.409262 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9a02d6-82e5-4f45-8b75-e94d9f0b17c5 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:36.409426 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9a02d6-82e5-4f45-8b75-e94d9f0b17c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:36.485787 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-8b9a02d6-82e5-4f45-8b75-e94d9f0b17c5 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:36.486998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-8b9a02d6-82e5-4f45-8b75-e94d9f0b17c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:36.554560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:36.583220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 723/1433] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 4 00:58:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:36.737180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea56bbb2-354b-4b70-8c37-652b1a1df3f1 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:36.739179 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ea56bbb2-354b-4b70-8c37-652b1a1df3f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:36.756600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:36.883963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea56bbb2-354b-4b70-8c37-652b1a1df3f1 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:36.970089 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:36.995175 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610bdbb4-a367-4c34-8c35-d5217ac87b27 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:36.995710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610bdbb4-a367-4c34-8c35-d5217ac87b27 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:36.995988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610bdbb4-a367-4c34-8c35-d5217ac87b27 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:36.996260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610bdbb4-a367-4c34-8c35-d5217ac87b27 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:36.998007 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-610bdbb4-a367-4c34-8c35-d5217ac87b27 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:37.091226 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610bdbb4-a367-4c34-8c35-d5217ac87b27 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:37.092807 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610bdbb4-a367-4c34-8c35-d5217ac87b27 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:37.110738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 711/1434] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:58:37.991568 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f34b678-8a21-414b-99e3-56fe2940f591 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:37.992153 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f34b678-8a21-414b-99e3-56fe2940f591 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:37.992153 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f34b678-8a21-414b-99e3-56fe2940f591 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:37.993516 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 724/1435] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 4 00:58:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:37.998463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf45ddf-434c-4480-bc82-cfb9913c2ca7 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:37.998881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf45ddf-434c-4480-bc82-cfb9913c2ca7 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:37.998989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf45ddf-434c-4480-bc82-cfb9913c2ca7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:38.074265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-edf45ddf-434c-4480-bc82-cfb9913c2ca7 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:38.075484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-edf45ddf-434c-4480-bc82-cfb9913c2ca7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:38.117884 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:38.173790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 712/1436] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 4 00:58:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:38.324944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f34b678-8a21-414b-99e3-56fe2940f591 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:38.326515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0f34b678-8a21-414b-99e3-56fe2940f591 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:38.471674 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f34b678-8a21-414b-99e3-56fe2940f591 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:38.525489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:38.583932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c6d291-e10a-43a2-936c-00a0cd5c4fac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:38.584466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c6d291-e10a-43a2-936c-00a0cd5c4fac service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:38.584743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c6d291-e10a-43a2-936c-00a0cd5c4fac service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:38.585643 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c6d291-e10a-43a2-936c-00a0cd5c4fac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:38.587253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-38c6d291-e10a-43a2-936c-00a0cd5c4fac service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:38.797161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c6d291-e10a-43a2-936c-00a0cd5c4fac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:38.798819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c6d291-e10a-43a2-936c-00a0cd5c4fac service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:38.818074 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 725/1437] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 494 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:58:39.690166 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd4efa4-1805-41ee-a7f6-a95e5b92b6fe None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:39.690719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd4efa4-1805-41ee-a7f6-a95e5b92b6fe None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:39.690719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd4efa4-1805-41ee-a7f6-a95e5b92b6fe None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:39.692856 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 713/1438] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 4 00:58:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:39.697813 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871a5c14-e613-41ea-8d33-31f9c9151884 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:39.698223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871a5c14-e613-41ea-8d33-31f9c9151884 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:39.698366 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871a5c14-e613-41ea-8d33-31f9c9151884 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:39.772175 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-871a5c14-e613-41ea-8d33-31f9c9151884 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:39.773323 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-871a5c14-e613-41ea-8d33-31f9c9151884 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:39.775351 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:39.869139 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 726/1439] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 4 00:58:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:40.023133 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd4efa4-1805-41ee-a7f6-a95e5b92b6fe None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:40.024542 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-1cd4efa4-1805-41ee-a7f6-a95e5b92b6fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:40.167462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd4efa4-1805-41ee-a7f6-a95e5b92b6fe None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:40.194499 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:40.278390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9a2ef1-7a07-4b68-8893-d0b6270f6375 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:40.278913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9a2ef1-7a07-4b68-8893-d0b6270f6375 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:40.279191 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9a2ef1-7a07-4b68-8893-d0b6270f6375 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:40.279462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9a2ef1-7a07-4b68-8893-d0b6270f6375 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:40.281127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-7d9a2ef1-7a07-4b68-8893-d0b6270f6375 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:40.370111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9a2ef1-7a07-4b68-8893-d0b6270f6375 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:40.371714 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9a2ef1-7a07-4b68-8893-d0b6270f6375 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:40.390070 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 714/1440] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:58:42.215932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79ffc65-18d7-401b-b83d-940d9c6bf5fb None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:42.216502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79ffc65-18d7-401b-b83d-940d9c6bf5fb None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:42.216502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79ffc65-18d7-401b-b83d-940d9c6bf5fb None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:42.217843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 727/1441] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 4 00:58:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:42.223677 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6389960e-7b91-44ad-9c97-c9e633ea49c4 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:42.224128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6389960e-7b91-44ad-9c97-c9e633ea49c4 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:42.224239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6389960e-7b91-44ad-9c97-c9e633ea49c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:42.265056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:42.290449 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-6389960e-7b91-44ad-9c97-c9e633ea49c4 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:42.291795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6389960e-7b91-44ad-9c97-c9e633ea49c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:42.388896 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 715/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 4 00:58:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:42.673586 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79ffc65-18d7-401b-b83d-940d9c6bf5fb None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:42.675227 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a79ffc65-18d7-401b-b83d-940d9c6bf5fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:42.779155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79ffc65-18d7-401b-b83d-940d9c6bf5fb None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:42.810269 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:42.852194 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31505616-cf44-4af9-af9c-8d0060152f2b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:42.852795 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31505616-cf44-4af9-af9c-8d0060152f2b service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:42.853081 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31505616-cf44-4af9-af9c-8d0060152f2b service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:42.853374 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31505616-cf44-4af9-af9c-8d0060152f2b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:42.854961 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-31505616-cf44-4af9-af9c-8d0060152f2b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:42.985426 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31505616-cf44-4af9-af9c-8d0060152f2b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:42.986853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31505616-cf44-4af9-af9c-8d0060152f2b service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:43.006347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 728/1443] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:58:45.079865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74dc3c1-7b2b-48e5-a73d-d750d1c8a003 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:45.080441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74dc3c1-7b2b-48e5-a73d-d750d1c8a003 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:45.080441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74dc3c1-7b2b-48e5-a73d-d750d1c8a003 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:45.082254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 716/1444] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 4 00:58:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:45.087482 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001cb212-03e9-445d-aa55-844bd5572ed2 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:45.087918 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001cb212-03e9-445d-aa55-844bd5572ed2 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:45.088026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001cb212-03e9-445d-aa55-844bd5572ed2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:45.153083 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-001cb212-03e9-445d-aa55-844bd5572ed2 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:45.154283 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-001cb212-03e9-445d-aa55-844bd5572ed2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:45.167089 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:45.249430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 729/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 4 00:58:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:45.535845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74dc3c1-7b2b-48e5-a73d-d750d1c8a003 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:45.537429 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-c74dc3c1-7b2b-48e5-a73d-d750d1c8a003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:45.639935 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74dc3c1-7b2b-48e5-a73d-d750d1c8a003 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:45.652327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:45.714089 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca10b8b7-bba6-45ed-a99e-a15b25abf26c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:45.714622 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca10b8b7-bba6-45ed-a99e-a15b25abf26c service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:45.714903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca10b8b7-bba6-45ed-a99e-a15b25abf26c service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:45.715191 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca10b8b7-bba6-45ed-a99e-a15b25abf26c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:45.716770 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ca10b8b7-bba6-45ed-a99e-a15b25abf26c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:45.824041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca10b8b7-bba6-45ed-a99e-a15b25abf26c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:45.825526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca10b8b7-bba6-45ed-a99e-a15b25abf26c service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:45.843987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 717/1446] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:58:48.121189 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8c08c7-b47b-4f31-9495-8b044800feb3 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:48.121766 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8c08c7-b47b-4f31-9495-8b044800feb3 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:48.121766 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8c08c7-b47b-4f31-9495-8b044800feb3 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:48.123115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 730/1447] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 4 00:58:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:48.128922 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e7b28d-5939-4f56-85fd-4fe7c656df0c None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:48.129379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e7b28d-5939-4f56-85fd-4fe7c656df0c None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:48.129513 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e7b28d-5939-4f56-85fd-4fe7c656df0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:48.176596 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:48.194776 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-59e7b28d-5939-4f56-85fd-4fe7c656df0c None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:48.195982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-59e7b28d-5939-4f56-85fd-4fe7c656df0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:48.291988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 718/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 4 00:58:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:48.574884 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8c08c7-b47b-4f31-9495-8b044800feb3 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:48.576442 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-5c8c08c7-b47b-4f31-9495-8b044800feb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:48.678967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8c08c7-b47b-4f31-9495-8b044800feb3 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:48.729672 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:48.763942 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856f5ac7-96ca-45b6-8ecf-fd43522d1889 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:48.764546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856f5ac7-96ca-45b6-8ecf-fd43522d1889 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:48.764827 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856f5ac7-96ca-45b6-8ecf-fd43522d1889 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:48.765122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856f5ac7-96ca-45b6-8ecf-fd43522d1889 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:48.766727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-856f5ac7-96ca-45b6-8ecf-fd43522d1889 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:48.876897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856f5ac7-96ca-45b6-8ecf-fd43522d1889 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:48.878298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856f5ac7-96ca-45b6-8ecf-fd43522d1889 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:48.897800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 731/1449] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:58:50.991078 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f055d150-1a02-42f9-9069-3950cc74107c None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:50.991650 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f055d150-1a02-42f9-9069-3950cc74107c None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:50.991650 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f055d150-1a02-42f9-9069-3950cc74107c None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:50.992981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 719/1450] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 4 00:58:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:50.998131 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f4de62-9f8c-4b4b-8013-dba3ce964dad None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:50.998552 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f4de62-9f8c-4b4b-8013-dba3ce964dad None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:50.998700 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f4de62-9f8c-4b4b-8013-dba3ce964dad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:51.063616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-83f4de62-9f8c-4b4b-8013-dba3ce964dad None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:51.064805 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-83f4de62-9f8c-4b4b-8013-dba3ce964dad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:51.086851 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:51.160861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 732/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 4 00:58:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:51.449346 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f055d150-1a02-42f9-9069-3950cc74107c None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:51.450933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f055d150-1a02-42f9-9069-3950cc74107c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:51.559441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f055d150-1a02-42f9-9069-3950cc74107c None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:51.579438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:51.633997 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c146ea-8a1e-4ba3-9e1b-c154aadf6384 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:51.634631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c146ea-8a1e-4ba3-9e1b-c154aadf6384 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:51.634912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c146ea-8a1e-4ba3-9e1b-c154aadf6384 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:51.635182 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c146ea-8a1e-4ba3-9e1b-c154aadf6384 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:51.636816 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-c1c146ea-8a1e-4ba3-9e1b-c154aadf6384 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:51.750017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c146ea-8a1e-4ba3-9e1b-c154aadf6384 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:51.751352 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c146ea-8a1e-4ba3-9e1b-c154aadf6384 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:51.769327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 720/1452] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:58:53.737975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe86b3c-6826-47a0-bd41-49eeaa5399ea None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:53.738539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe86b3c-6826-47a0-bd41-49eeaa5399ea None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:53.738539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe86b3c-6826-47a0-bd41-49eeaa5399ea None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:53.739910 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 733/1453] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 4 00:58:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:53.745363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e799b01-5863-46cb-9f7a-cad900c09bc1 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:53.745762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e799b01-5863-46cb-9f7a-cad900c09bc1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:53.745916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e799b01-5863-46cb-9f7a-cad900c09bc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:53.808838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:53.811841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-8e799b01-5863-46cb-9f7a-cad900c09bc1 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:53.813059 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8e799b01-5863-46cb-9f7a-cad900c09bc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:53.912180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 721/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 4 00:58:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:54.198660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe86b3c-6826-47a0-bd41-49eeaa5399ea None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:54.200191 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-3fe86b3c-6826-47a0-bd41-49eeaa5399ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:54.302981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe86b3c-6826-47a0-bd41-49eeaa5399ea None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:54.350954 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:54.375667 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb26f73c-9c64-40eb-a0f2-e4432deffda2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:54.376259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb26f73c-9c64-40eb-a0f2-e4432deffda2 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:54.376534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb26f73c-9c64-40eb-a0f2-e4432deffda2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:54.376802 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb26f73c-9c64-40eb-a0f2-e4432deffda2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:54.378997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-bb26f73c-9c64-40eb-a0f2-e4432deffda2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:54.485500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb26f73c-9c64-40eb-a0f2-e4432deffda2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:54.487448 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb26f73c-9c64-40eb-a0f2-e4432deffda2 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:54.506422 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 734/1455] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:58:56.492346 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836db36-f7b2-41ef-9f9e-02fe019f5adf None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:56.492818 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836db36-f7b2-41ef-9f9e-02fe019f5adf None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:56.492914 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836db36-f7b2-41ef-9f9e-02fe019f5adf None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:56.494250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 722/1456] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 4 00:58:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:58:56.499513 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1b861a-da56-4d57-a4bf-3f67c8b38b7f None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:56.499915 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1b861a-da56-4d57-a4bf-3f67c8b38b7f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:56.500095 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df1b861a-da56-4d57-a4bf-3f67c8b38b7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:56.566871 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-df1b861a-da56-4d57-a4bf-3f67c8b38b7f None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:56.568074 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-df1b861a-da56-4d57-a4bf-3f67c8b38b7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:56.597885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:56.669157 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 735/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 4 00:58:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:56.956294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3836db36-f7b2-41ef-9f9e-02fe019f5adf None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:56.958380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-3836db36-f7b2-41ef-9f9e-02fe019f5adf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:57.060523 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3836db36-f7b2-41ef-9f9e-02fe019f5adf None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:57.089809 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:57.137846 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4517220-0d40-4978-8ad5-ca9dc6c1907e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:57.138447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4517220-0d40-4978-8ad5-ca9dc6c1907e service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:57.138732 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4517220-0d40-4978-8ad5-ca9dc6c1907e service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:57.138994 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4517220-0d40-4978-8ad5-ca9dc6c1907e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:57.140574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f4517220-0d40-4978-8ad5-ca9dc6c1907e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:57.246800 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4517220-0d40-4978-8ad5-ca9dc6c1907e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:58:57.248143 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4517220-0d40-4978-8ad5-ca9dc6c1907e service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:58:57.266981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 723/1458] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:58:59.304870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0202b708-1d20-4e79-b8b9-0fadd24a84b8 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:59.305477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0202b708-1d20-4e79-b8b9-0fadd24a84b8 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:59.305477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0202b708-1d20-4e79-b8b9-0fadd24a84b8 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:59.306875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 736/1459] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 4 00:58:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:58:59.312300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688296ce-c397-481d-a06f-b07591062ab9 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:59.312680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688296ce-c397-481d-a06f-b07591062ab9 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:59.312842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688296ce-c397-481d-a06f-b07591062ab9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:59.377993 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-688296ce-c397-481d-a06f-b07591062ab9 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:58:59.379184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-688296ce-c397-481d-a06f-b07591062ab9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:59.387277 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:59.476423 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 724/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 4 00:58:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:58:59.764223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0202b708-1d20-4e79-b8b9-0fadd24a84b8 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:58:59.765814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0202b708-1d20-4e79-b8b9-0fadd24a84b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:58:59.867329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0202b708-1d20-4e79-b8b9-0fadd24a84b8 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:58:59.921140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:58:59.940637 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffa55a7-eef6-4970-b3a8-68afa6cf45c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:58:59.941277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa55a7-eef6-4970-b3a8-68afa6cf45c1 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:58:59.941559 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa55a7-eef6-4970-b3a8-68afa6cf45c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:58:59.941834 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa55a7-eef6-4970-b3a8-68afa6cf45c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:58:59.943435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7ffa55a7-eef6-4970-b3a8-68afa6cf45c1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:00.050589 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffa55a7-eef6-4970-b3a8-68afa6cf45c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:59:00.052013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffa55a7-eef6-4970-b3a8-68afa6cf45c1 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:59:00.069756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 737/1461] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:58:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:59:02.063335 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508647bb-e5c1-48b9-8fe3-d4dbf9a1f843 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:02.063868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508647bb-e5c1-48b9-8fe3-d4dbf9a1f843 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:02.063868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508647bb-e5c1-48b9-8fe3-d4dbf9a1f843 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:02.065859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 725/1462] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 4 00:59:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 00:59:02.071107 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4debb3e1-7af2-4e93-bafe-2de944721110 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:02.071561 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4debb3e1-7af2-4e93-bafe-2de944721110 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:02.071661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4debb3e1-7af2-4e93-bafe-2de944721110 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:02.138842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-4debb3e1-7af2-4e93-bafe-2de944721110 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:59:02.139976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4debb3e1-7af2-4e93-bafe-2de944721110 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:02.177400 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:02.234335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 738/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 4 00:59:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:59:02.526573 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508647bb-e5c1-48b9-8fe3-d4dbf9a1f843 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:59:02.528091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-508647bb-e5c1-48b9-8fe3-d4dbf9a1f843 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:02.630736 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508647bb-e5c1-48b9-8fe3-d4dbf9a1f843 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:59:02.668193 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:02.703393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f404db7-333d-4b47-aad0-2c4520ecfa10 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:59:02.703997 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f404db7-333d-4b47-aad0-2c4520ecfa10 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:02.704277 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f404db7-333d-4b47-aad0-2c4520ecfa10 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:02.704547 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f404db7-333d-4b47-aad0-2c4520ecfa10 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:02.706845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-7f404db7-333d-4b47-aad0-2c4520ecfa10 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:02.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f404db7-333d-4b47-aad0-2c4520ecfa10 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:59:02.821491 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f404db7-333d-4b47-aad0-2c4520ecfa10 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:59:02.840327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 726/1464] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:59:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:59:04.787611 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc104492-d755-41aa-9198-6018ce8efa66 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:04.787611 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc104492-d755-41aa-9198-6018ce8efa66 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:04.787611 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc104492-d755-41aa-9198-6018ce8efa66 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:04.787611 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 739/1465] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 4 00:59:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:59:04.788584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c52633f-ac07-4a15-8af9-0e79629baf4b None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:04.788584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c52633f-ac07-4a15-8af9-0e79629baf4b None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:04.789133 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c52633f-ac07-4a15-8af9-0e79629baf4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:04.861506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-7c52633f-ac07-4a15-8af9-0e79629baf4b None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:59:04.862990 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-7c52633f-ac07-4a15-8af9-0e79629baf4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:04.884294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:04.980102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 727/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 4 00:59:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:59:05.304525 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc104492-d755-41aa-9198-6018ce8efa66 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:59:05.306967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-fc104492-d755-41aa-9198-6018ce8efa66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:05.428605 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc104492-d755-41aa-9198-6018ce8efa66 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:59:05.509596 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:05.516281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19871821-f91e-4566-83e0-a7e4977499e2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:59:05.516891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19871821-f91e-4566-83e0-a7e4977499e2 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:05.517187 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19871821-f91e-4566-83e0-a7e4977499e2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:05.517515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19871821-f91e-4566-83e0-a7e4977499e2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:05.520517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-19871821-f91e-4566-83e0-a7e4977499e2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:05.659880 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19871821-f91e-4566-83e0-a7e4977499e2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:59:05.661622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19871821-f91e-4566-83e0-a7e4977499e2 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:59:05.680176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 740/1467] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:59:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:59:07.614291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39a5add-2eed-4756-8c79-09d0aafdd7da None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:07.616525 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39a5add-2eed-4756-8c79-09d0aafdd7da None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:07.616525 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39a5add-2eed-4756-8c79-09d0aafdd7da None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:07.616525 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 728/1468] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 4 00:59:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:59:07.621965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746e2bf8-0bc1-4480-8a7e-3de15b680ce9 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:07.622438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746e2bf8-0bc1-4480-8a7e-3de15b680ce9 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:07.622550 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746e2bf8-0bc1-4480-8a7e-3de15b680ce9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:07.688334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-746e2bf8-0bc1-4480-8a7e-3de15b680ce9 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:59:07.690102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-746e2bf8-0bc1-4480-8a7e-3de15b680ce9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:07.737550 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:07.794966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 741/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 4 00:59:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:59:08.079185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39a5add-2eed-4756-8c79-09d0aafdd7da None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:59:08.081290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a39a5add-2eed-4756-8c79-09d0aafdd7da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:08.184835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39a5add-2eed-4756-8c79-09d0aafdd7da None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:59:08.231558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:08.257310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dced416d-194e-4db9-86b5-645fca54b274 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:59:08.257908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dced416d-194e-4db9-86b5-645fca54b274 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:08.258183 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dced416d-194e-4db9-86b5-645fca54b274 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:08.258450 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dced416d-194e-4db9-86b5-645fca54b274 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:08.260018 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-dced416d-194e-4db9-86b5-645fca54b274 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:08.365877 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dced416d-194e-4db9-86b5-645fca54b274 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:59:08.367262 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dced416d-194e-4db9-86b5-645fca54b274 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:59:08.385962 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 729/1470] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:59:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:59:17.793121 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=22 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:18.383681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=66 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:28.087030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66239b26-119c-4ba9-9db9-0a2af0bb67d5 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:28.087571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66239b26-119c-4ba9-9db9-0a2af0bb67d5 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:28.087571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66239b26-119c-4ba9-9db9-0a2af0bb67d5 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:28.088901 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 742/1471] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:59:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:59:28.094089 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d73ac4-6945-4852-83c6-6a04fc087658 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:28.095082 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d73ac4-6945-4852-83c6-6a04fc087658 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:28.095365 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d73ac4-6945-4852-83c6-6a04fc087658 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:28.159947 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-b3d73ac4-6945-4852-83c6-6a04fc087658 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:59:28.161173 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b3d73ac4-6945-4852-83c6-6a04fc087658 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:28.254664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 730/1472] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:59:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:59:28.550934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7d8108-ef9b-48d9-a887-564e7c368c9d None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:28.551209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7d8108-ef9b-48d9-a887-564e7c368c9d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:28.551474 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7d8108-ef9b-48d9-a887-564e7c368c9d None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:28.552784 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 743/1473] 10.222.0.42 () {54 vars in 961 bytes} [Sun Sep 4 00:59:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:59:28.557669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9355b8-5829-42a0-ae74-7918be79f057 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:28.557957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9355b8-5829-42a0-ae74-7918be79f057 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:28.558257 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9355b8-5829-42a0-ae74-7918be79f057 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:28.635078 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-6b9355b8-5829-42a0-ae74-7918be79f057 None None] MFA Rules not processed for user `69153344fc4b45588bdbcecdfcfd7c3f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:59:28.636296 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6b9355b8-5829-42a0-ae74-7918be79f057 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:28.656392 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:28.712289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 731/1474] 10.222.0.42 () {58 vars in 1058 bytes} [Sun Sep 4 00:59:28 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:59:28.720510 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7d8108-ef9b-48d9-a887-564e7c368c9d None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:59:28.722005 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-cb7d8108-ef9b-48d9-a887-564e7c368c9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:28.826594 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7d8108-ef9b-48d9-a887-564e7c368c9d None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:59:28.914544 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47243b8b-359d-473e-9563-0a2d201ce949 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '69153344fc4b45588bdbcecdfcfd7c3f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:59:28.915184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47243b8b-359d-473e-9563-0a2d201ce949 service glance] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:28.915467 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47243b8b-359d-473e-9563-0a2d201ce949 service glance] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:28.915739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47243b8b-359d-473e-9563-0a2d201ce949 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:28.917376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-47243b8b-359d-473e-9563-0a2d201ce949 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:28.973260 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:29.029778 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47243b8b-359d-473e-9563-0a2d201ce949 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:59:29.031120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47243b8b-359d-473e-9563-0a2d201ce949 service glance] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:59:29.049519 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 744/1475] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:59:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:59:31.035776 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0dc77f-f4e0-4cbc-bd3d-fea58fc858ee None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:31.036348 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0dc77f-f4e0-4cbc-bd3d-fea58fc858ee None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:31.036348 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0dc77f-f4e0-4cbc-bd3d-fea58fc858ee None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:31.038409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 732/1476] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:59:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:59:31.043171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8961918-834a-4faa-8ed1-958e280e076e None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:31.043591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8961918-834a-4faa-8ed1-958e280e076e None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:31.043745 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8961918-834a-4faa-8ed1-958e280e076e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:31.109051 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-a8961918-834a-4faa-8ed1-958e280e076e None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:59:31.110823 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a8961918-834a-4faa-8ed1-958e280e076e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:31.207319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 745/1477] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:59:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:59:31.484012 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0dc77f-f4e0-4cbc-bd3d-fea58fc858ee None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:59:31.485603 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-7a0dc77f-f4e0-4cbc-bd3d-fea58fc858ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:31.611187 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0dc77f-f4e0-4cbc-bd3d-fea58fc858ee None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:59:31.659760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:31.705892 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e751e021-379e-4b69-bd16-f208490b186e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:59:31.706498 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e751e021-379e-4b69-bd16-f208490b186e admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:31.706797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e751e021-379e-4b69-bd16-f208490b186e admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:31.707062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e751e021-379e-4b69-bd16-f208490b186e admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:31.707701 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e751e021-379e-4b69-bd16-f208490b186e admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:59:31.709455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: WARNING py.warnings [None req-e751e021-379e-4b69-bd16-f208490b186e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 04 00:59:31.709455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: warnings.warn(msg) Sep 04 00:59:31.709455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: Sep 04 00:59:31.710609 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e751e021-379e-4b69-bd16-f208490b186e admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:59:31.729091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 733/1478] 10.222.0.42 () {56 vars in 1108 bytes} [Sun Sep 4 00:59:31 2022] GET /identity/v3/projects => generated 2553 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 00:59:33.638381 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78052cdb-3078-4e68-8700-bf1dc132dee4 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:33.638917 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78052cdb-3078-4e68-8700-bf1dc132dee4 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:33.639014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78052cdb-3078-4e68-8700-bf1dc132dee4 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:33.640296 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 746/1479] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:59:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:59:33.645934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b1b5da-3a4a-444e-9c35-e2a4828272c8 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:33.646339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b1b5da-3a4a-444e-9c35-e2a4828272c8 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:33.646487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b1b5da-3a4a-444e-9c35-e2a4828272c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:33.712833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-d4b1b5da-3a4a-444e-9c35-e2a4828272c8 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:59:33.713986 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d4b1b5da-3a4a-444e-9c35-e2a4828272c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:33.809202 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 734/1480] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:59:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 00:59:34.105321 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78052cdb-3078-4e68-8700-bf1dc132dee4 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:59:34.109426 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-78052cdb-3078-4e68-8700-bf1dc132dee4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:34.117422 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:34.253410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78052cdb-3078-4e68-8700-bf1dc132dee4 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:59:34.331495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:34.364089 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbec2868-e67e-4878-b4df-d579a4ab6fce service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:59:34.364626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbec2868-e67e-4878-b4df-d579a4ab6fce service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:34.364904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbec2868-e67e-4878-b4df-d579a4ab6fce service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:34.365206 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbec2868-e67e-4878-b4df-d579a4ab6fce service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:34.367050 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-cbec2868-e67e-4878-b4df-d579a4ab6fce service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:34.454957 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbec2868-e67e-4878-b4df-d579a4ab6fce service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:59:34.456542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbec2868-e67e-4878-b4df-d579a4ab6fce service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:59:34.475215 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 747/1481] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:59:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:59:36.566209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b448eec-16bb-45be-aea8-1ad3f9f42239 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:36.566804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b448eec-16bb-45be-aea8-1ad3f9f42239 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:36.566804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b448eec-16bb-45be-aea8-1ad3f9f42239 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:36.568125 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 735/1482] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:59:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:59:36.573373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c458dc3-b88c-484c-ba73-a7ab33dc18a8 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:36.573786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c458dc3-b88c-484c-ba73-a7ab33dc18a8 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:36.573919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c458dc3-b88c-484c-ba73-a7ab33dc18a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:36.638384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-7c458dc3-b88c-484c-ba73-a7ab33dc18a8 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:59:36.639573 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7c458dc3-b88c-484c-ba73-a7ab33dc18a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:36.686099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:36.735878 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 748/1483] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:59:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:59:37.023694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b448eec-16bb-45be-aea8-1ad3f9f42239 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:59:37.025287 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8b448eec-16bb-45be-aea8-1ad3f9f42239 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:37.049574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:37.129258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b448eec-16bb-45be-aea8-1ad3f9f42239 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:59:37.199940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9fe5d8-41c9-4224-9136-760ab573abaa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:59:37.200537 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9fe5d8-41c9-4224-9136-760ab573abaa service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:37.200814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9fe5d8-41c9-4224-9136-760ab573abaa service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:37.201104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9fe5d8-41c9-4224-9136-760ab573abaa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:37.202689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-9c9fe5d8-41c9-4224-9136-760ab573abaa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:37.274991 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:37.308976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9fe5d8-41c9-4224-9136-760ab573abaa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:59:37.310435 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9fe5d8-41c9-4224-9136-760ab573abaa service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:59:37.329518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 736/1484] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:59:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:59:39.343387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fc1b8d-a369-492f-84e9-1b6383ebce23 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:39.344477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fc1b8d-a369-492f-84e9-1b6383ebce23 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:39.344593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fc1b8d-a369-492f-84e9-1b6383ebce23 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:39.345937 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 749/1485] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:59:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:59:39.351152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e962ae9-2560-4488-a310-3865dcf617ae None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:39.351596 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e962ae9-2560-4488-a310-3865dcf617ae None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:39.351702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e962ae9-2560-4488-a310-3865dcf617ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:39.417204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-4e962ae9-2560-4488-a310-3865dcf617ae None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:59:39.418401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-4e962ae9-2560-4488-a310-3865dcf617ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:39.514966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 737/1486] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:59:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:59:39.813567 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fc1b8d-a369-492f-84e9-1b6383ebce23 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:59:39.815117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-34fc1b8d-a369-492f-84e9-1b6383ebce23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:39.919271 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fc1b8d-a369-492f-84e9-1b6383ebce23 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:59:39.989057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:39.991225 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906205bf-76c8-4643-97ed-4089829edc9f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:59:39.991824 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906205bf-76c8-4643-97ed-4089829edc9f service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:39.992097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906205bf-76c8-4643-97ed-4089829edc9f service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:39.992371 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906205bf-76c8-4643-97ed-4089829edc9f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:39.993979 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-906205bf-76c8-4643-97ed-4089829edc9f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:40.101328 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906205bf-76c8-4643-97ed-4089829edc9f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:59:40.102765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906205bf-76c8-4643-97ed-4089829edc9f service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:59:40.121145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 750/1487] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:59:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:59:42.223155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad547ff-af39-4ef4-8e44-f1b69dbdcf4f None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:42.223724 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad547ff-af39-4ef4-8e44-f1b69dbdcf4f None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:42.223724 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad547ff-af39-4ef4-8e44-f1b69dbdcf4f None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:42.225102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 738/1488] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:59:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:59:42.230483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6074e20a-7943-4ba7-819b-ea3514f129aa None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:42.230910 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6074e20a-7943-4ba7-819b-ea3514f129aa None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:42.231037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6074e20a-7943-4ba7-819b-ea3514f129aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:42.323865 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-6074e20a-7943-4ba7-819b-ea3514f129aa None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:59:42.325105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-6074e20a-7943-4ba7-819b-ea3514f129aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:42.377716 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:42.420902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 751/1489] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:59:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:59:42.728309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad547ff-af39-4ef4-8e44-f1b69dbdcf4f None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:59:42.729890 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-bad547ff-af39-4ef4-8e44-f1b69dbdcf4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:42.766095 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:42.837398 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad547ff-af39-4ef4-8e44-f1b69dbdcf4f None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:59:42.914994 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf00426-aa29-4a77-ac0f-28f06b212403 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:59:42.915619 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf00426-aa29-4a77-ac0f-28f06b212403 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:42.915906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf00426-aa29-4a77-ac0f-28f06b212403 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:42.916181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf00426-aa29-4a77-ac0f-28f06b212403 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:42.918060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-4bf00426-aa29-4a77-ac0f-28f06b212403 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:43.014520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:43.043170 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf00426-aa29-4a77-ac0f-28f06b212403 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:59:43.044641 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf00426-aa29-4a77-ac0f-28f06b212403 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:59:43.064268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 739/1490] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:59:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:59:45.418069 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6ac3a1-8844-470b-bf36-544cfd8370dd None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:45.418611 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6ac3a1-8844-470b-bf36-544cfd8370dd None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:45.418611 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6ac3a1-8844-470b-bf36-544cfd8370dd None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:45.419991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 752/1491] 10.222.0.42 () {54 vars in 927 bytes} [Sun Sep 4 00:59:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 00:59:45.425396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7b0fb0-d74b-43ad-a55d-c71290cbea54 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:45.425817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7b0fb0-d74b-43ad-a55d-c71290cbea54 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:45.425939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7b0fb0-d74b-43ad-a55d-c71290cbea54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:45.495780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-cb7b0fb0-d74b-43ad-a55d-c71290cbea54 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 00:59:45.497001 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-cb7b0fb0-d74b-43ad-a55d-c71290cbea54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:45.593285 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 740/1492] 10.222.0.42 () {58 vars in 1024 bytes} [Sun Sep 4 00:59:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 00:59:45.889294 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6ac3a1-8844-470b-bf36-544cfd8370dd None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 00:59:45.889294 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9c6ac3a1-8844-470b-bf36-544cfd8370dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:45.991040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6ac3a1-8844-470b-bf36-544cfd8370dd None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 00:59:46.063238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc954c71-3895-4c97-9872-e6fee81efb67 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 00:59:46.063846 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc954c71-3895-4c97-9872-e6fee81efb67 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 00:59:46.064129 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc954c71-3895-4c97-9872-e6fee81efb67 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 00:59:46.064404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc954c71-3895-4c97-9872-e6fee81efb67 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 00:59:46.066044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-dc954c71-3895-4c97-9872-e6fee81efb67 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 00:59:46.075956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:46.174951 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc954c71-3895-4c97-9872-e6fee81efb67 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 00:59:46.176309 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc954c71-3895-4c97-9872-e6fee81efb67 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 00:59:46.194161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 753/1493] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 00:59:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 00:59:55.591446 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=82 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 00:59:56.191747 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=50 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:01:30.917464 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41877438-52d6-4532-a8a1-95d6624c4488 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:01:30.918728 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41877438-52d6-4532-a8a1-95d6624c4488 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:01:30.918728 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41877438-52d6-4532-a8a1-95d6624c4488 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:01:30.994927 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-41877438-52d6-4532-a8a1-95d6624c4488 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:01:30.996182 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-41877438-52d6-4532-a8a1-95d6624c4488 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:01:31.092662 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 741/1494] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:01:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:01:31.100556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415d86b5-4b31-478c-a0ad-807955a38b49 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:01:31.101039 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415d86b5-4b31-478c-a0ad-807955a38b49 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:01:31.101162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415d86b5-4b31-478c-a0ad-807955a38b49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:01:31.189787 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-415d86b5-4b31-478c-a0ad-807955a38b49 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:01:31.190933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-415d86b5-4b31-478c-a0ad-807955a38b49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:01:31.286021 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 754/1495] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:01:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:01:31.290655 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41877438-52d6-4532-a8a1-95d6624c4488 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:01:31.291837 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-41877438-52d6-4532-a8a1-95d6624c4488 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:01:31.302944 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41877438-52d6-4532-a8a1-95d6624c4488 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:01:31.362445 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:01:31.395862 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465db36b-c710-4072-b8f3-1ccb7a938926 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:01:31.396528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465db36b-c710-4072-b8f3-1ccb7a938926 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:01:31.396819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465db36b-c710-4072-b8f3-1ccb7a938926 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:01:31.397121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465db36b-c710-4072-b8f3-1ccb7a938926 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:01:31.397985 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465db36b-c710-4072-b8f3-1ccb7a938926 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:01:31.398457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465db36b-c710-4072-b8f3-1ccb7a938926 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:01:31.399744 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: WARNING py.warnings [None req-465db36b-c710-4072-b8f3-1ccb7a938926 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 04 01:01:31.399744 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: warnings.warn(msg) Sep 04 01:01:31.399744 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: Sep 04 01:01:31.400626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465db36b-c710-4072-b8f3-1ccb7a938926 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:01:31.411538 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 742/1496] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:01:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:01:31.420142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415d86b5-4b31-478c-a0ad-807955a38b49 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:01:31.429263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415d86b5-4b31-478c-a0ad-807955a38b49 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:01:31.490861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:01:31.525037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af481ac-60a9-4133-a9e6-49a7ad060388 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:01:31.525671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af481ac-60a9-4133-a9e6-49a7ad060388 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:01:31.525947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af481ac-60a9-4133-a9e6-49a7ad060388 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:01:31.526213 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af481ac-60a9-4133-a9e6-49a7ad060388 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:01:31.526831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af481ac-60a9-4133-a9e6-49a7ad060388 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:01:31.528608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: WARNING py.warnings [None req-2af481ac-60a9-4133-a9e6-49a7ad060388 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 04 01:01:31.528608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: warnings.warn(msg) Sep 04 01:01:31.528608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: Sep 04 01:01:31.529816 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af481ac-60a9-4133-a9e6-49a7ad060388 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:01:31.656541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 755/1497] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:01:31 2022] POST /identity/v3/projects => generated 379 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:01:31.660746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465db36b-c710-4072-b8f3-1ccb7a938926 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:01:31.662322 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-465db36b-c710-4072-b8f3-1ccb7a938926 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:01:31.695871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465db36b-c710-4072-b8f3-1ccb7a938926 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:01:31.794316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8845fa-6efd-4281-a771-54653bafe92e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:01:31.794831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8845fa-6efd-4281-a771-54653bafe92e admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:01:31.795108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8845fa-6efd-4281-a771-54653bafe92e admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:01:31.795378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8845fa-6efd-4281-a771-54653bafe92e admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:01:31.796034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8845fa-6efd-4281-a771-54653bafe92e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:01:31.797010 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: WARNING py.warnings [None req-1d8845fa-6efd-4281-a771-54653bafe92e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 04 01:01:31.797010 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: warnings.warn(msg) Sep 04 01:01:31.797010 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: Sep 04 01:01:31.798255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8845fa-6efd-4281-a771-54653bafe92e admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:01:31.895378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 743/1498] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:01:31 2022] POST /identity/v3/users => generated 290 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:01:31.899558 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af481ac-60a9-4133-a9e6-49a7ad060388 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:01:31.901134 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-2af481ac-60a9-4133-a9e6-49a7ad060388 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:01:31.965517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af481ac-60a9-4133-a9e6-49a7ad060388 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:01:32.049662 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:01:32.060358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84644417-912d-471b-8cbc-24bdbefbba82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:01:32.060958 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84644417-912d-471b-8cbc-24bdbefbba82 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:01:32.061264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84644417-912d-471b-8cbc-24bdbefbba82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:01:32.061533 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84644417-912d-471b-8cbc-24bdbefbba82 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:01:32.062173 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84644417-912d-471b-8cbc-24bdbefbba82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:01:32.063178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: WARNING py.warnings [None req-84644417-912d-471b-8cbc-24bdbefbba82 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 04 01:01:32.063178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: warnings.warn(msg) Sep 04 01:01:32.063178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: Sep 04 01:01:32.064059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84644417-912d-471b-8cbc-24bdbefbba82 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:01:32.075956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 756/1499] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:01:31 2022] GET /identity/v3/roles => generated 1420 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:01:32.080164 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8845fa-6efd-4281-a771-54653bafe92e admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:01:32.081699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-1d8845fa-6efd-4281-a771-54653bafe92e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:01:32.093258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8845fa-6efd-4281-a771-54653bafe92e admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:01:32.118690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:01:32.193986 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3ad38e-202c-43a9-88bd-8288c996f1b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:01:32.194597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3ad38e-202c-43a9-88bd-8288c996f1b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:01:32.194884 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3ad38e-202c-43a9-88bd-8288c996f1b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:01:32.195187 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3ad38e-202c-43a9-88bd-8288c996f1b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:01:32.195845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3ad38e-202c-43a9-88bd-8288c996f1b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:01:32.196860 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: WARNING py.warnings [None req-6e3ad38e-202c-43a9-88bd-8288c996f1b8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 04 01:01:32.196860 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: warnings.warn(msg) Sep 04 01:01:32.196860 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: Sep 04 01:01:32.197801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3ad38e-202c-43a9-88bd-8288c996f1b8 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:01:32.214013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 744/1500] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:01:32 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:01:32.219400 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84644417-912d-471b-8cbc-24bdbefbba82 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:01:32.220932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-84644417-912d-471b-8cbc-24bdbefbba82 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:01:32.254839 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84644417-912d-471b-8cbc-24bdbefbba82 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:01:32.347407 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa827dc-be0a-4bf7-9f11-288c5143d7f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:01:32.348056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa827dc-be0a-4bf7-9f11-288c5143d7f0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:01:32.348335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa827dc-be0a-4bf7-9f11-288c5143d7f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:01:32.348630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa827dc-be0a-4bf7-9f11-288c5143d7f0 admin admin] PATH_INFO: `/v3/projects/e7e4ee55a7b245f5b2670340ddb85c47/users/990153ce94ee464488fd20c51b9bc0ad/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:01:32.370410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa827dc-be0a-4bf7-9f11-288c5143d7f0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e7e4ee55a7b245f5b2670340ddb85c47, user_id=990153ce94ee464488fd20c51b9bc0ad, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:01:32.371550 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: WARNING py.warnings [None req-7aa827dc-be0a-4bf7-9f11-288c5143d7f0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 04 01:01:32.371550 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: warnings.warn(msg) Sep 04 01:01:32.371550 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: Sep 04 01:01:32.373498 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa827dc-be0a-4bf7-9f11-288c5143d7f0 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:01:32.539525 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 757/1501] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:01:32 2022] PUT /identity/v3/projects/e7e4ee55a7b245f5b2670340ddb85c47/users/990153ce94ee464488fd20c51b9bc0ad/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:01:32.552080 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f839cced-0b83-4844-a18a-a4369187555e None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:01:32.552436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f839cced-0b83-4844-a18a-a4369187555e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:01:32.552661 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f839cced-0b83-4844-a18a-a4369187555e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:01:32.614325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-f839cced-0b83-4844-a18a-a4369187555e None None] MFA Rules not processed for user `990153ce94ee464488fd20c51b9bc0ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:01:32.615451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f839cced-0b83-4844-a18a-a4369187555e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:01:32.692316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 745/1502] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:01:32 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:01:32.705914 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0519dfad-f7ae-4b75-9b56-916bb70a8749 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:01:32.706199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0519dfad-f7ae-4b75-9b56-916bb70a8749 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:01:32.706471 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0519dfad-f7ae-4b75-9b56-916bb70a8749 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:01:32.707853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 758/1503] 10.222.0.42 () {54 vars in 964 bytes} [Sun Sep 4 01:01:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:01:32.712614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa8ea90-6c86-4c4d-b564-fc68dd112f46 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:01:32.712897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa8ea90-6c86-4c4d-b564-fc68dd112f46 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:01:32.713204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa8ea90-6c86-4c4d-b564-fc68dd112f46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:01:32.745230 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:01:32.789550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-6aa8ea90-6c86-4c4d-b564-fc68dd112f46 None None] MFA Rules not processed for user `615a607b6be14b82979f94eabaf4c34c`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:01:32.790764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6aa8ea90-6c86-4c4d-b564-fc68dd112f46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:01:32.865706 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 746/1504] 10.222.0.42 () {58 vars in 1061 bytes} [Sun Sep 4 01:01:32 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:01:32.874169 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0519dfad-f7ae-4b75-9b56-916bb70a8749 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:01:32.875668 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0519dfad-f7ae-4b75-9b56-916bb70a8749 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:01:32.927089 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:01:32.981200 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0519dfad-f7ae-4b75-9b56-916bb70a8749 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:01:33.054728 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f6895e-6f00-4002-93ba-a4e04267e522 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:01:33.055329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f6895e-6f00-4002-93ba-a4e04267e522 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:01:33.055605 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f6895e-6f00-4002-93ba-a4e04267e522 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:01:33.055875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f6895e-6f00-4002-93ba-a4e04267e522 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:01:33.057494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-31f6895e-6f00-4002-93ba-a4e04267e522 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:01:33.156719 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f6895e-6f00-4002-93ba-a4e04267e522 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:01:33.158115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f6895e-6f00-4002-93ba-a4e04267e522 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:01:33.161546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:01:33.176512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 759/1505] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:01:32 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:01:33.222832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9d44bd-1fdf-44be-b8ff-041f915ddd6b None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:01:33.223247 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9d44bd-1fdf-44be-b8ff-041f915ddd6b None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:01:33.223394 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9d44bd-1fdf-44be-b8ff-041f915ddd6b None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:01:33.224640 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 747/1506] 10.222.0.42 () {52 vars in 830 bytes} [Sun Sep 4 01:01:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:01:33.228316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f6895e-6f00-4002-93ba-a4e04267e522 service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:01:33.229560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-31f6895e-6f00-4002-93ba-a4e04267e522 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:01:33.329661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f6895e-6f00-4002-93ba-a4e04267e522 service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:01:33.422557 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2b3d3c-25c0-4a5b-9533-6ff7fa9d04c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:01:33.423106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2b3d3c-25c0-4a5b-9533-6ff7fa9d04c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:01:33.423386 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2b3d3c-25c0-4a5b-9533-6ff7fa9d04c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:01:33.423661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2b3d3c-25c0-4a5b-9533-6ff7fa9d04c6 admin admin] PATH_INFO: `/v3/users/990153ce94ee464488fd20c51b9bc0ad` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:01:33.437564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2b3d3c-25c0-4a5b-9533-6ff7fa9d04c6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=990153ce94ee464488fd20c51b9bc0ad)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:01:33.438686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: WARNING py.warnings [None req-7c2b3d3c-25c0-4a5b-9533-6ff7fa9d04c6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 04 01:01:33.438686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: warnings.warn(msg) Sep 04 01:01:33.438686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: Sep 04 01:01:33.439906 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2b3d3c-25c0-4a5b-9533-6ff7fa9d04c6 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:01:33.450059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:01:33.651345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7c2b3d3c-25c0-4a5b-9533-6ff7fa9d04c6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '990153ce94ee464488fd20c51b9bc0ad'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:01:33.743146 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7c2b3d3c-25c0-4a5b-9533-6ff7fa9d04c6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '990153ce94ee464488fd20c51b9bc0ad'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:01:33.767403 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7c2b3d3c-25c0-4a5b-9533-6ff7fa9d04c6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '990153ce94ee464488fd20c51b9bc0ad'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:01:33.767983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7c2b3d3c-25c0-4a5b-9533-6ff7fa9d04c6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '990153ce94ee464488fd20c51b9bc0ad'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:01:33.782467 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7c2b3d3c-25c0-4a5b-9533-6ff7fa9d04c6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '990153ce94ee464488fd20c51b9bc0ad'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:01:33.789647 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 760/1507] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:01:33 2022] DELETE /identity/v3/users/990153ce94ee464488fd20c51b9bc0ad => generated 0 bytes in 562 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:01:33.794724 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080944c8-e7bc-42e0-9fdc-bc2f2a362492 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:01:33.795014 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080944c8-e7bc-42e0-9fdc-bc2f2a362492 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:01:33.795296 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080944c8-e7bc-42e0-9fdc-bc2f2a362492 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:01:33.852550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-080944c8-e7bc-42e0-9fdc-bc2f2a362492 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:01:33.853787 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-080944c8-e7bc-42e0-9fdc-bc2f2a362492 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:01:33.929158 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:01:33.941136 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 748/1508] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:01:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:01:33.950824 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2b3d3c-25c0-4a5b-9533-6ff7fa9d04c6 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:01:33.952176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7c2b3d3c-25c0-4a5b-9533-6ff7fa9d04c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:01:34.057007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2b3d3c-25c0-4a5b-9533-6ff7fa9d04c6 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:01:34.090958 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c4a8d5-f3c8-43b6-b23e-839483cf4f35 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:01:34.091565 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c4a8d5-f3c8-43b6-b23e-839483cf4f35 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:01:34.091849 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c4a8d5-f3c8-43b6-b23e-839483cf4f35 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:01:34.092118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c4a8d5-f3c8-43b6-b23e-839483cf4f35 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:01:34.094165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d1c4a8d5-f3c8-43b6-b23e-839483cf4f35 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:01:34.125303 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:01:34.213386 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c4a8d5-f3c8-43b6-b23e-839483cf4f35 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:01:34.215472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c4a8d5-f3c8-43b6-b23e-839483cf4f35 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:01:34.235133 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 761/1509] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:01:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:01:34.751369 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080944c8-e7bc-42e0-9fdc-bc2f2a362492 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:01:34.751369 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-080944c8-e7bc-42e0-9fdc-bc2f2a362492 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:01:34.764033 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080944c8-e7bc-42e0-9fdc-bc2f2a362492 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:01:34.784823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b468003-0032-4edb-938d-cae39e50967a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:01:34.785557 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b468003-0032-4edb-938d-cae39e50967a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:01:34.785834 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b468003-0032-4edb-938d-cae39e50967a admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:01:34.786125 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b468003-0032-4edb-938d-cae39e50967a admin admin] PATH_INFO: `/v3/projects/e7e4ee55a7b245f5b2670340ddb85c47` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:01:34.809048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b468003-0032-4edb-938d-cae39e50967a admin admin] RBAC: Authorizing `identity:delete_project(project_id=e7e4ee55a7b245f5b2670340ddb85c47)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:01:34.809048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: WARNING py.warnings [None req-8b468003-0032-4edb-938d-cae39e50967a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 04 01:01:34.809048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: warnings.warn(msg) Sep 04 01:01:34.809048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: Sep 04 01:01:34.809048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b468003-0032-4edb-938d-cae39e50967a admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:01:34.931649 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8b468003-0032-4edb-938d-cae39e50967a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e7e4ee55a7b245f5b2670340ddb85c47', 'request_id': 'req-8b468003-0032-4edb-938d-cae39e50967a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:01:34.937705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8b468003-0032-4edb-938d-cae39e50967a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e7e4ee55a7b245f5b2670340ddb85c47', 'request_id': 'req-8b468003-0032-4edb-938d-cae39e50967a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:01:35.004058 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8b468003-0032-4edb-938d-cae39e50967a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e7e4ee55a7b245f5b2670340ddb85c47', 'request_id': 'req-8b468003-0032-4edb-938d-cae39e50967a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:01:35.023164 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8b468003-0032-4edb-938d-cae39e50967a admin admin] The token cache is being invalidate because project e7e4ee55a7b245f5b2670340ddb85c47 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:01:35.023459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8b468003-0032-4edb-938d-cae39e50967a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:01:35.024224 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8b468003-0032-4edb-938d-cae39e50967a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:01:35.025746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 749/1510] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:01:34 2022] DELETE /identity/v3/projects/e7e4ee55a7b245f5b2670340ddb85c47 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:01:44.238364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=41 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:01:45.020238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=40,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:03:00.875957 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6c6d21-2c5a-4701-bddd-d93fe437bd43 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:00.876568 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6c6d21-2c5a-4701-bddd-d93fe437bd43 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:00.876568 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6c6d21-2c5a-4701-bddd-d93fe437bd43 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:00.877871 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 762/1511] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 4 01:03:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:03:00.882755 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5214d8-d084-47c4-aa8c-c0df8df288b5 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:00.883206 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5214d8-d084-47c4-aa8c-c0df8df288b5 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:00.883324 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5214d8-d084-47c4-aa8c-c0df8df288b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:00.941837 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-4d5214d8-d084-47c4-aa8c-c0df8df288b5 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:03:00.943037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-4d5214d8-d084-47c4-aa8c-c0df8df288b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:03:01.000732 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 750/1512] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 4 01:03:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:03:01.158862 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6c6d21-2c5a-4701-bddd-d93fe437bd43 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:03:01.160626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9a6c6d21-2c5a-4701-bddd-d93fe437bd43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:03:01.264654 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6c6d21-2c5a-4701-bddd-d93fe437bd43 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:03:01.270997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42e098a-3493-476a-b2df-c1e8aaa3288b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:03:01.271627 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42e098a-3493-476a-b2df-c1e8aaa3288b service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:01.271902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42e098a-3493-476a-b2df-c1e8aaa3288b service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:01.272171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42e098a-3493-476a-b2df-c1e8aaa3288b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:01.281162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42e098a-3493-476a-b2df-c1e8aaa3288b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:03:01.282760 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42e098a-3493-476a-b2df-c1e8aaa3288b service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:03:01.292133 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 763/1513] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:03:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:03:02.117515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9bd3de-3a7c-4f51-827f-49b1d6444bd4 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:02.118056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9bd3de-3a7c-4f51-827f-49b1d6444bd4 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:02.118056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9bd3de-3a7c-4f51-827f-49b1d6444bd4 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:02.119497 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 751/1514] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 4 01:03:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:03:02.124526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941c8052-53d0-4999-a8da-8e3f3ed3a444 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:02.124967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941c8052-53d0-4999-a8da-8e3f3ed3a444 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:02.125061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941c8052-53d0-4999-a8da-8e3f3ed3a444 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:02.144342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-941c8052-53d0-4999-a8da-8e3f3ed3a444 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:03:02.145582 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-941c8052-53d0-4999-a8da-8e3f3ed3a444 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:03:02.153919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 764/1515] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 4 01:03:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:03:02.305928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9bd3de-3a7c-4f51-827f-49b1d6444bd4 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:03:02.320000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9bd3de-3a7c-4f51-827f-49b1d6444bd4 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:03:02.325661 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acd0078-862a-4461-990c-7bed5275b942 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:03:02.326253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acd0078-862a-4461-990c-7bed5275b942 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:02.326533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acd0078-862a-4461-990c-7bed5275b942 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:02.326811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acd0078-862a-4461-990c-7bed5275b942 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:02.334930 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acd0078-862a-4461-990c-7bed5275b942 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:03:02.336493 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acd0078-862a-4461-990c-7bed5275b942 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:03:02.346549 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 752/1516] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:03:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:03:02.599129 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180154ab-6a8b-4dec-be34-cb2dd0ee2d9c None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:02.599412 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180154ab-6a8b-4dec-be34-cb2dd0ee2d9c None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:02.599684 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180154ab-6a8b-4dec-be34-cb2dd0ee2d9c None None] PATH_INFO: `/v3` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:02.601061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 765/1517] 10.222.0.16 () {54 vars in 925 bytes} [Sun Sep 4 01:03:02 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:03:02.608325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e42aaf-9bfa-4a80-8011-9d8ce0dd1df6 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:02.608606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e42aaf-9bfa-4a80-8011-9d8ce0dd1df6 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:02.608877 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e42aaf-9bfa-4a80-8011-9d8ce0dd1df6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:02.668604 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-74e42aaf-9bfa-4a80-8011-9d8ce0dd1df6 None None] MFA Rules not processed for user `b29f444d56ec49308f7fef16f4d1eea5`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:03:02.669758 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-74e42aaf-9bfa-4a80-8011-9d8ce0dd1df6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:03:02.741384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 753/1518] 10.222.0.16 () {58 vars in 1015 bytes} [Sun Sep 4 01:03:02 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:03:11.845591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebd437c-9aea-40ff-9481-6bb971b5f26a None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:11.846154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebd437c-9aea-40ff-9481-6bb971b5f26a None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:11.846154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebd437c-9aea-40ff-9481-6bb971b5f26a None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:11.847469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 766/1519] 10.222.0.42 () {54 vars in 961 bytes} [Sun Sep 4 01:03:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:03:11.852325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89057c0e-6007-41bc-becb-16655034b0b5 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:11.852735 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89057c0e-6007-41bc-becb-16655034b0b5 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:11.852874 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89057c0e-6007-41bc-becb-16655034b0b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:11.872716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-89057c0e-6007-41bc-becb-16655034b0b5 None None] MFA Rules not processed for user `b29f444d56ec49308f7fef16f4d1eea5`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:03:11.873933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-89057c0e-6007-41bc-becb-16655034b0b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:03:11.882227 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 754/1520] 10.222.0.42 () {58 vars in 1058 bytes} [Sun Sep 4 01:03:11 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:03:11.890120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebd437c-9aea-40ff-9481-6bb971b5f26a None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:03:11.903476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebd437c-9aea-40ff-9481-6bb971b5f26a None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:03:11.908342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc9df64-4c1a-45a0-a88d-8ad219e50a82 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b29f444d56ec49308f7fef16f4d1eea5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:03:11.908920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc9df64-4c1a-45a0-a88d-8ad219e50a82 service placement] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:11.909229 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc9df64-4c1a-45a0-a88d-8ad219e50a82 service placement] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:11.909501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc9df64-4c1a-45a0-a88d-8ad219e50a82 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:11.917757 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc9df64-4c1a-45a0-a88d-8ad219e50a82 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:03:11.919326 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc9df64-4c1a-45a0-a88d-8ad219e50a82 service placement] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:03:11.928666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 767/1521] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:03:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:03:13.540219 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb653f6a-f990-4e66-a56f-5cdedd578dd3 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:13.541501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb653f6a-f990-4e66-a56f-5cdedd578dd3 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:13.541501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb653f6a-f990-4e66-a56f-5cdedd578dd3 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:13.542857 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 755/1522] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 4 01:03:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:03:13.547629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9285b7-af08-457e-b5d3-a412a019775f None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:13.548082 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9285b7-af08-457e-b5d3-a412a019775f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:13.548190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9285b7-af08-457e-b5d3-a412a019775f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:13.567389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9f9285b7-af08-457e-b5d3-a412a019775f None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:03:13.568592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9f9285b7-af08-457e-b5d3-a412a019775f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:03:13.576429 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 768/1523] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 4 01:03:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:03:13.730417 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb653f6a-f990-4e66-a56f-5cdedd578dd3 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:03:13.739977 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:03:13.745215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb653f6a-f990-4e66-a56f-5cdedd578dd3 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:03:13.750823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64341b94-21c8-4df3-8377-27993a114f8d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:03:13.751424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64341b94-21c8-4df3-8377-27993a114f8d service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:13.751703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64341b94-21c8-4df3-8377-27993a114f8d service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:13.751978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64341b94-21c8-4df3-8377-27993a114f8d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:13.760810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64341b94-21c8-4df3-8377-27993a114f8d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:03:13.762477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64341b94-21c8-4df3-8377-27993a114f8d service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:03:13.771662 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 756/1524] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:03:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:03:14.605933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bae9ee-1db9-43f0-b560-b22f5e085753 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:14.606497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bae9ee-1db9-43f0-b560-b22f5e085753 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:14.606497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bae9ee-1db9-43f0-b560-b22f5e085753 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:14.607855 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 769/1525] 127.0.0.1 () {54 vars in 901 bytes} [Sun Sep 4 01:03:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:03:14.613090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fdf8c6-f0d4-47fd-9570-e009d4674d68 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:14.613487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fdf8c6-f0d4-47fd-9570-e009d4674d68 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:14.613665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fdf8c6-f0d4-47fd-9570-e009d4674d68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:14.632789 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-b1fdf8c6-f0d4-47fd-9570-e009d4674d68 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:03:14.634044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b1fdf8c6-f0d4-47fd-9570-e009d4674d68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:03:14.642478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 757/1526] 127.0.0.1 () {58 vars in 998 bytes} [Sun Sep 4 01:03:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:03:14.796695 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bae9ee-1db9-43f0-b560-b22f5e085753 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:03:14.811601 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bae9ee-1db9-43f0-b560-b22f5e085753 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:03:14.817053 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49278c5-fc67-4cdf-a05a-c10318199f29 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:03:14.817667 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49278c5-fc67-4cdf-a05a-c10318199f29 service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:14.817942 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49278c5-fc67-4cdf-a05a-c10318199f29 service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:14.818213 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49278c5-fc67-4cdf-a05a-c10318199f29 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:14.827261 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49278c5-fc67-4cdf-a05a-c10318199f29 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:03:14.832231 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49278c5-fc67-4cdf-a05a-c10318199f29 service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:03:14.838607 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 770/1527] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:03:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:03:18.815795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd81d243-7cd1-48e6-b6e4-01a60dda692a None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:18.816340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd81d243-7cd1-48e6-b6e4-01a60dda692a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:18.816453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd81d243-7cd1-48e6-b6e4-01a60dda692a None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:18.817797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 758/1528] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 4 01:03:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:03:18.823707 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434f2c83-a288-4b29-adf2-68a0716064f2 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:18.824134 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434f2c83-a288-4b29-adf2-68a0716064f2 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:18.824248 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434f2c83-a288-4b29-adf2-68a0716064f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:18.843886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-434f2c83-a288-4b29-adf2-68a0716064f2 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:03:18.845098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-434f2c83-a288-4b29-adf2-68a0716064f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:03:18.852873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 771/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 4 01:03:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:03:19.139182 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd81d243-7cd1-48e6-b6e4-01a60dda692a None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:03:19.140892 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-dd81d243-7cd1-48e6-b6e4-01a60dda692a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:03:19.219683 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd81d243-7cd1-48e6-b6e4-01a60dda692a None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:03:19.223237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3696688a-cc1d-4483-bfc2-17ac66ce79a2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:03:19.223838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3696688a-cc1d-4483-bfc2-17ac66ce79a2 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:19.224114 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3696688a-cc1d-4483-bfc2-17ac66ce79a2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:19.224381 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3696688a-cc1d-4483-bfc2-17ac66ce79a2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:19.233241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3696688a-cc1d-4483-bfc2-17ac66ce79a2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:03:19.234653 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3696688a-cc1d-4483-bfc2-17ac66ce79a2 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:03:19.244046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 759/1530] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:03:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:03:21.135394 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5014006e-c73a-4d04-beab-f4c03dd649f6 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:21.135969 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5014006e-c73a-4d04-beab-f4c03dd649f6 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:21.136078 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5014006e-c73a-4d04-beab-f4c03dd649f6 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:21.137433 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 772/1531] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 4 01:03:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:03:21.142500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86a83dc-5415-453d-ac7a-d2641a232092 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:21.142949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86a83dc-5415-453d-ac7a-d2641a232092 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:21.143059 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86a83dc-5415-453d-ac7a-d2641a232092 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:21.162320 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-f86a83dc-5415-453d-ac7a-d2641a232092 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:03:21.163508 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f86a83dc-5415-453d-ac7a-d2641a232092 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:03:21.171885 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 760/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 4 01:03:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:03:21.455788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5014006e-c73a-4d04-beab-f4c03dd649f6 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:03:21.466816 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5014006e-c73a-4d04-beab-f4c03dd649f6 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:03:21.470409 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859452b3-9d05-4aeb-b8da-16d07a022187 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:03:21.471008 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859452b3-9d05-4aeb-b8da-16d07a022187 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:21.471285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859452b3-9d05-4aeb-b8da-16d07a022187 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:21.471554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859452b3-9d05-4aeb-b8da-16d07a022187 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:21.480935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859452b3-9d05-4aeb-b8da-16d07a022187 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:03:21.482399 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859452b3-9d05-4aeb-b8da-16d07a022187 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:03:21.491818 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 773/1533] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:03:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:03:24.904287 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193263cd-11b5-494f-986a-d8ca2e6536e0 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:24.904852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193263cd-11b5-494f-986a-d8ca2e6536e0 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:24.904852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193263cd-11b5-494f-986a-d8ca2e6536e0 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:24.906200 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 761/1534] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 4 01:03:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:03:24.911244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc8c477-1a4d-4555-b3e3-7561d3190c65 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:24.911685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc8c477-1a4d-4555-b3e3-7561d3190c65 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:24.911827 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc8c477-1a4d-4555-b3e3-7561d3190c65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:24.920257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=74 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:03:24.932787 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9dc8c477-1a4d-4555-b3e3-7561d3190c65 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:03:24.934016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9dc8c477-1a4d-4555-b3e3-7561d3190c65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:03:24.941813 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 774/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 4 01:03:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:03:25.500614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193263cd-11b5-494f-986a-d8ca2e6536e0 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:03:25.502453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-193263cd-11b5-494f-986a-d8ca2e6536e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:03:25.581920 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193263cd-11b5-494f-986a-d8ca2e6536e0 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:03:25.585492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73a9e18-f741-4aa0-9cc8-f162bf2cb4af service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:03:25.586092 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73a9e18-f741-4aa0-9cc8-f162bf2cb4af service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:25.586370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73a9e18-f741-4aa0-9cc8-f162bf2cb4af service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:25.586641 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73a9e18-f741-4aa0-9cc8-f162bf2cb4af service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:25.596025 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73a9e18-f741-4aa0-9cc8-f162bf2cb4af service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:03:25.597467 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73a9e18-f741-4aa0-9cc8-f162bf2cb4af service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:03:25.607255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 762/1536] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:03:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:03:27.522870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ba179a-9fd6-4163-b331-db4e5ca8172c None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:27.523380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ba179a-9fd6-4163-b331-db4e5ca8172c None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:27.523498 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ba179a-9fd6-4163-b331-db4e5ca8172c None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:27.525314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 775/1537] 127.0.0.1 () {54 vars in 917 bytes} [Sun Sep 4 01:03:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:03:27.530610 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623c5242-f68d-44ad-a700-db135dd99b88 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:27.531017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623c5242-f68d-44ad-a700-db135dd99b88 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:27.531160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623c5242-f68d-44ad-a700-db135dd99b88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:27.551294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-623c5242-f68d-44ad-a700-db135dd99b88 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:03:27.552500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-623c5242-f68d-44ad-a700-db135dd99b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:03:27.560748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 763/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Sep 4 01:03:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:03:28.121931 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ba179a-9fd6-4163-b331-db4e5ca8172c None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:03:28.123635 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a7ba179a-9fd6-4163-b331-db4e5ca8172c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:03:28.136800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ba179a-9fd6-4163-b331-db4e5ca8172c None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:03:28.140405 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c68421-4c99-4913-adb3-74c93022fb81 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:03:28.140992 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c68421-4c99-4913-adb3-74c93022fb81 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:03:28.141302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c68421-4c99-4913-adb3-74c93022fb81 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:03:28.141576 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c68421-4c99-4913-adb3-74c93022fb81 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:03:28.150571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c68421-4c99-4913-adb3-74c93022fb81 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:03:28.151991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c68421-4c99-4913-adb3-74c93022fb81 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:03:28.161316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 776/1539] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:03:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:03:37.546457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=83 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:03:38.157193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=8 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:50.393405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca69ecc8-42a9-44d9-a651-74ba79735bbb None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:50.476039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca69ecc8-42a9-44d9-a651-74ba79735bbb None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:50.476039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca69ecc8-42a9-44d9-a651-74ba79735bbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:50.476039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-ca69ecc8-42a9-44d9-a651-74ba79735bbb None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:50.476039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ca69ecc8-42a9-44d9-a651-74ba79735bbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:50.476039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 764/1540] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:04:50.478727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d6f17a-dfa8-4925-bc23-51772f517ba1 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:50.478727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d6f17a-dfa8-4925-bc23-51772f517ba1 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:50.478727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d6f17a-dfa8-4925-bc23-51772f517ba1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:50.498108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-30d6f17a-dfa8-4925-bc23-51772f517ba1 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:50.499456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-30d6f17a-dfa8-4925-bc23-51772f517ba1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:50.511095 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 777/1541] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:50.516602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca69ecc8-42a9-44d9-a651-74ba79735bbb None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:50.531534 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca69ecc8-42a9-44d9-a651-74ba79735bbb None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:50.541306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccf87ea-ab0d-4268-87d8-76fa2c14c7d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:50.541952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccf87ea-ab0d-4268-87d8-76fa2c14c7d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:50.542226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bdfe7d-69d7-4e0d-8f3c-d723988e1388 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:50.542349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccf87ea-ab0d-4268-87d8-76fa2c14c7d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:50.542525 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccf87ea-ab0d-4268-87d8-76fa2c14c7d9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:50.542654 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bdfe7d-69d7-4e0d-8f3c-d723988e1388 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:50.542748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bdfe7d-69d7-4e0d-8f3c-d723988e1388 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:50.543164 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccf87ea-ab0d-4268-87d8-76fa2c14c7d9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:04:50.543638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccf87ea-ab0d-4268-87d8-76fa2c14c7d9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:50.546577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccf87ea-ab0d-4268-87d8-76fa2c14c7d9 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:50.563261 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 765/1542] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:04:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:04:50.572195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccf87ea-ab0d-4268-87d8-76fa2c14c7d9 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:50.589101 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-07bdfe7d-69d7-4e0d-8f3c-d723988e1388 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:50.589101 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-07bdfe7d-69d7-4e0d-8f3c-d723988e1388 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:50.595238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccf87ea-ab0d-4268-87d8-76fa2c14c7d9 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:50.606232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9e98f9-285b-4deb-8945-a83bd6473955 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:50.606863 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9e98f9-285b-4deb-8945-a83bd6473955 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:50.607145 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9e98f9-285b-4deb-8945-a83bd6473955 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:50.607422 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9e98f9-285b-4deb-8945-a83bd6473955 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:50.608071 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9e98f9-285b-4deb-8945-a83bd6473955 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:50.609111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: WARNING py.warnings [None req-4a9e98f9-285b-4deb-8945-a83bd6473955 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 04 01:04:50.609111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: warnings.warn(msg) Sep 04 01:04:50.609111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: Sep 04 01:04:50.610470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9e98f9-285b-4deb-8945-a83bd6473955 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:50.614229 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 778/1543] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:04:50.633317 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677218ab-f6fe-4491-851a-cb7439c45cbe None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:50.633577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677218ab-f6fe-4491-851a-cb7439c45cbe None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:50.633911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677218ab-f6fe-4491-851a-cb7439c45cbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:50.676315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-677218ab-f6fe-4491-851a-cb7439c45cbe None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:50.676315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-677218ab-f6fe-4491-851a-cb7439c45cbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:50.682927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 779/1544] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:04:50.687310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677218ab-f6fe-4491-851a-cb7439c45cbe None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:50.696917 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677218ab-f6fe-4491-851a-cb7439c45cbe None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:50.705678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b9113f-73de-43ff-ad8d-e71100137fcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:50.706634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b9113f-73de-43ff-ad8d-e71100137fcd admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:50.707092 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b9113f-73de-43ff-ad8d-e71100137fcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:50.707539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b9113f-73de-43ff-ad8d-e71100137fcd admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:50.708440 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b9113f-73de-43ff-ad8d-e71100137fcd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:04:50.709885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b9113f-73de-43ff-ad8d-e71100137fcd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:50.711313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: WARNING py.warnings [None req-94b9113f-73de-43ff-ad8d-e71100137fcd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 04 01:04:50.711313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: warnings.warn(msg) Sep 04 01:04:50.711313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: Sep 04 01:04:50.716906 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b9113f-73de-43ff-ad8d-e71100137fcd admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:50.745062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 766/1545] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:04:50 2022] POST /identity/v3/projects => generated 379 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:50.748268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9e98f9-285b-4deb-8945-a83bd6473955 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:50.751101 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 780/1546] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:04:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 01:04:50.761048 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b9113f-73de-43ff-ad8d-e71100137fcd admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:50.780989 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b9113f-73de-43ff-ad8d-e71100137fcd admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:50.874020 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9e98f9-285b-4deb-8945-a83bd6473955 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:50.889069 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5420f323-acb9-41fb-a206-829761db510d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:50.889069 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5420f323-acb9-41fb-a206-829761db510d admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:50.889069 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5420f323-acb9-41fb-a206-829761db510d admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:50.889069 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5420f323-acb9-41fb-a206-829761db510d admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:50.897149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5420f323-acb9-41fb-a206-829761db510d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:50.897149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5420f323-acb9-41fb-a206-829761db510d admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:50.910932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a42237f-4653-41dc-94e0-d95161fe91f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:50.912894 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a42237f-4653-41dc-94e0-d95161fe91f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:50.916223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a42237f-4653-41dc-94e0-d95161fe91f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:50.916855 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a42237f-4653-41dc-94e0-d95161fe91f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:50.917852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a42237f-4653-41dc-94e0-d95161fe91f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:50.920797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a42237f-4653-41dc-94e0-d95161fe91f9 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:51.014139 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 767/1547] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:04:50 2022] POST /identity/v3/users => generated 290 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:51.015615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f138f24-6d0a-43a9-a0e8-ac64f85aaec1 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.015900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f138f24-6d0a-43a9-a0e8-ac64f85aaec1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.016173 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f138f24-6d0a-43a9-a0e8-ac64f85aaec1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.045339 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 781/1548] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:04:50 2022] POST /identity/v3/projects => generated 385 bytes in 285 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:04:51.047015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-3f138f24-6d0a-43a9-a0e8-ac64f85aaec1 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:51.047015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-3f138f24-6d0a-43a9-a0e8-ac64f85aaec1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:51.049396 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a42237f-4653-41dc-94e0-d95161fe91f9 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:51.168072 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 768/1549] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:51.168730 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f138f24-6d0a-43a9-a0e8-ac64f85aaec1 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:51.184094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a42237f-4653-41dc-94e0-d95161fe91f9 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:51.188252 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f138f24-6d0a-43a9-a0e8-ac64f85aaec1 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:51.195690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61aab0de-64ee-441b-8433-8851dd7112e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:51.201495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aab0de-64ee-441b-8433-8851dd7112e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.201495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aab0de-64ee-441b-8433-8851dd7112e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.201495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aab0de-64ee-441b-8433-8851dd7112e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.201495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61aab0de-64ee-441b-8433-8851dd7112e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:51.201495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61aab0de-64ee-441b-8433-8851dd7112e5 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:51.206337 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71c629d-6477-48f4-ad7f-4563257e15fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:51.206968 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71c629d-6477-48f4-ad7f-4563257e15fa admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.207247 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71c629d-6477-48f4-ad7f-4563257e15fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.207517 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71c629d-6477-48f4-ad7f-4563257e15fa admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.208160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71c629d-6477-48f4-ad7f-4563257e15fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:51.210988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71c629d-6477-48f4-ad7f-4563257e15fa admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:51.220576 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 782/1550] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:51 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:51.221883 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0687002-4fad-4d33-87d5-61c4260a8ce9 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.222172 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0687002-4fad-4d33-87d5-61c4260a8ce9 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.222456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0687002-4fad-4d33-87d5-61c4260a8ce9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.264883 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-c0687002-4fad-4d33-87d5-61c4260a8ce9 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:51.266072 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c0687002-4fad-4d33-87d5-61c4260a8ce9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:51.280433 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 783/1551] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:51.281820 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9e2e2f-f7b4-45e4-8f55-42d594747701 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.282302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9e2e2f-f7b4-45e4-8f55-42d594747701 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.282573 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9e2e2f-f7b4-45e4-8f55-42d594747701 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.303344 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-ed9e2e2f-f7b4-45e4-8f55-42d594747701 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:51.304560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ed9e2e2f-f7b4-45e4-8f55-42d594747701 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:51.313015 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 784/1552] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:51.313828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9e2e2f-f7b4-45e4-8f55-42d594747701 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:51.325869 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed9e2e2f-f7b4-45e4-8f55-42d594747701 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:51.330608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903d873b-7141-489f-b078-33b66fa8b009 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:51.331192 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903d873b-7141-489f-b078-33b66fa8b009 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.331467 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903d873b-7141-489f-b078-33b66fa8b009 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.331736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903d873b-7141-489f-b078-33b66fa8b009 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.332374 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903d873b-7141-489f-b078-33b66fa8b009 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:51.334619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903d873b-7141-489f-b078-33b66fa8b009 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:51.344727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 785/1553] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:51.345878 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a00b844-8a76-42f1-95bd-470d9eaff48d None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.346158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a00b844-8a76-42f1-95bd-470d9eaff48d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.346427 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a00b844-8a76-42f1-95bd-470d9eaff48d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.365514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-7a00b844-8a76-42f1-95bd-470d9eaff48d None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:51.366768 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7a00b844-8a76-42f1-95bd-470d9eaff48d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:51.374590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 786/1554] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:51.375698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173584af-1709-46fa-833e-bb521eae4dcf None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.375979 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173584af-1709-46fa-833e-bb521eae4dcf None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.376248 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173584af-1709-46fa-833e-bb521eae4dcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.481281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-173584af-1709-46fa-833e-bb521eae4dcf None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:51.483950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-173584af-1709-46fa-833e-bb521eae4dcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:51.492700 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 787/1555] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:51.493929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dba2db9-5fac-42ae-a53a-3476df841288 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.494216 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dba2db9-5fac-42ae-a53a-3476df841288 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.494493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dba2db9-5fac-42ae-a53a-3476df841288 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.517755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9dba2db9-5fac-42ae-a53a-3476df841288 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:51.523470 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9dba2db9-5fac-42ae-a53a-3476df841288 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:51.527075 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 788/1556] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:51.528289 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60345b93-8c4a-47a6-908e-970291e0b0ae None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.528588 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60345b93-8c4a-47a6-908e-970291e0b0ae None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.529922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60345b93-8c4a-47a6-908e-970291e0b0ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.538501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:51.539321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 769/1557] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:04:51 2022] POST /identity/v3/users => generated 293 bytes in 371 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:51.543213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71c629d-6477-48f4-ad7f-4563257e15fa admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:51.553958 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-60345b93-8c4a-47a6-908e-970291e0b0ae None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:51.554719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71c629d-6477-48f4-ad7f-4563257e15fa admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:51.555421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-60345b93-8c4a-47a6-908e-970291e0b0ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:51.559582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a49b1f7-30c6-490d-94a8-91dde2d8f6d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:51.560328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a49b1f7-30c6-490d-94a8-91dde2d8f6d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.560781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a49b1f7-30c6-490d-94a8-91dde2d8f6d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.561134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a49b1f7-30c6-490d-94a8-91dde2d8f6d7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.561836 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a49b1f7-30c6-490d-94a8-91dde2d8f6d7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:04:51.562101 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a49b1f7-30c6-490d-94a8-91dde2d8f6d7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:51.563801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a49b1f7-30c6-490d-94a8-91dde2d8f6d7 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:51.569413 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 789/1558] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:51.576531 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114ee3e3-57ef-4aeb-bc79-3fb10f54ab95 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.583141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114ee3e3-57ef-4aeb-bc79-3fb10f54ab95 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.583141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114ee3e3-57ef-4aeb-bc79-3fb10f54ab95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.583369 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 770/1559] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:04:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:04:51.583369 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a49b1f7-30c6-490d-94a8-91dde2d8f6d7 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:51.592712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a49b1f7-30c6-490d-94a8-91dde2d8f6d7 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:51.597758 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae0e20b-449c-4778-ae7c-f7682d4cdc13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:51.598507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae0e20b-449c-4778-ae7c-f7682d4cdc13 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.598861 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae0e20b-449c-4778-ae7c-f7682d4cdc13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.599253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae0e20b-449c-4778-ae7c-f7682d4cdc13 admin admin] PATH_INFO: `/v3/projects/e07d8b14fef34e04a0410f12e5dbd2fe/users/b6b95b6814cf4cd8b77d87e7810c70e5/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.602022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-114ee3e3-57ef-4aeb-bc79-3fb10f54ab95 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:51.604524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:51.605197 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-114ee3e3-57ef-4aeb-bc79-3fb10f54ab95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:51.615431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 790/1560] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:51.616970 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7c377c-94da-4da1-b5d0-53d9cc33d307 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.617432 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7c377c-94da-4da1-b5d0-53d9cc33d307 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.617909 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7c377c-94da-4da1-b5d0-53d9cc33d307 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.625584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae0e20b-449c-4778-ae7c-f7682d4cdc13 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e07d8b14fef34e04a0410f12e5dbd2fe, user_id=b6b95b6814cf4cd8b77d87e7810c70e5, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:51.627716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: WARNING py.warnings [None req-1ae0e20b-449c-4778-ae7c-f7682d4cdc13 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 04 01:04:51.627716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: warnings.warn(msg) Sep 04 01:04:51.627716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: Sep 04 01:04:51.631311 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae0e20b-449c-4778-ae7c-f7682d4cdc13 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:51.641185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-6e7c377c-94da-4da1-b5d0-53d9cc33d307 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:51.642682 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-6e7c377c-94da-4da1-b5d0-53d9cc33d307 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:51.651033 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 791/1561] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:51.652569 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eef7ad2-eff0-47b5-b631-37140faf479d None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.652995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eef7ad2-eff0-47b5-b631-37140faf479d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.653541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eef7ad2-eff0-47b5-b631-37140faf479d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.673917 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-5eef7ad2-eff0-47b5-b631-37140faf479d None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:51.675168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-5eef7ad2-eff0-47b5-b631-37140faf479d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:51.684370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 792/1562] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:51.685395 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85462ff6-0b90-4469-9eed-2fe3f83ae042 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.685678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85462ff6-0b90-4469-9eed-2fe3f83ae042 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.685950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85462ff6-0b90-4469-9eed-2fe3f83ae042 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.707387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-85462ff6-0b90-4469-9eed-2fe3f83ae042 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:51.708653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-85462ff6-0b90-4469-9eed-2fe3f83ae042 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:51.716602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 793/1563] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:51.717400 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85462ff6-0b90-4469-9eed-2fe3f83ae042 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:51.726658 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85462ff6-0b90-4469-9eed-2fe3f83ae042 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:51.731353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb40bdca-5a25-4d10-b78e-62b9d706c2a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:51.731953 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb40bdca-5a25-4d10-b78e-62b9d706c2a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.732233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb40bdca-5a25-4d10-b78e-62b9d706c2a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.732502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb40bdca-5a25-4d10-b78e-62b9d706c2a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.733233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb40bdca-5a25-4d10-b78e-62b9d706c2a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:51.734835 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb40bdca-5a25-4d10-b78e-62b9d706c2a4 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:51.746606 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 794/1564] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:51 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:51.756463 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb40bdca-5a25-4d10-b78e-62b9d706c2a4 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:51.760957 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb40bdca-5a25-4d10-b78e-62b9d706c2a4 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:51.765840 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d97bc5c-3b96-47a8-b3aa-6dccc1f1a4c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:51.766436 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d97bc5c-3b96-47a8-b3aa-6dccc1f1a4c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.766714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d97bc5c-3b96-47a8-b3aa-6dccc1f1a4c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.766986 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d97bc5c-3b96-47a8-b3aa-6dccc1f1a4c5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.767631 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d97bc5c-3b96-47a8-b3aa-6dccc1f1a4c5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:04:51.767979 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d97bc5c-3b96-47a8-b3aa-6dccc1f1a4c5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:51.769560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d97bc5c-3b96-47a8-b3aa-6dccc1f1a4c5 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:51.783353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 795/1565] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:04:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:04:51.784117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d97bc5c-3b96-47a8-b3aa-6dccc1f1a4c5 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:51.797927 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 771/1566] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:04:51 2022] PUT /identity/v3/projects/e07d8b14fef34e04a0410f12e5dbd2fe/users/b6b95b6814cf4cd8b77d87e7810c70e5/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:04:51.799493 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7cde59-e116-4e05-b133-7b656e102456 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.799642 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d97bc5c-3b96-47a8-b3aa-6dccc1f1a4c5 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:51.799800 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7cde59-e116-4e05-b133-7b656e102456 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.800046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7cde59-e116-4e05-b133-7b656e102456 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.831645 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-ea7cde59-e116-4e05-b133-7b656e102456 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:51.832869 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ea7cde59-e116-4e05-b133-7b656e102456 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:51.862275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171436f5-d382-4a2c-98ca-c2d457dfb579 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:51.862908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171436f5-d382-4a2c-98ca-c2d457dfb579 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.863188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171436f5-d382-4a2c-98ca-c2d457dfb579 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.863461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171436f5-d382-4a2c-98ca-c2d457dfb579 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.864096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171436f5-d382-4a2c-98ca-c2d457dfb579 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:51.867042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171436f5-d382-4a2c-98ca-c2d457dfb579 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:51.898538 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 772/1567] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:51.899394 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7cde59-e116-4e05-b133-7b656e102456 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:51.908337 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7cde59-e116-4e05-b133-7b656e102456 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:51.914051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9f4337-6892-4adf-8420-a201728544c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:51.914635 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9f4337-6892-4adf-8420-a201728544c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.914919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9f4337-6892-4adf-8420-a201728544c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.915193 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9f4337-6892-4adf-8420-a201728544c0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.915841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9f4337-6892-4adf-8420-a201728544c0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:04:51.916147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9f4337-6892-4adf-8420-a201728544c0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:51.917813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9f4337-6892-4adf-8420-a201728544c0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:51.930042 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 773/1568] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:04:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:04:51.930951 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9f4337-6892-4adf-8420-a201728544c0 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:51.940388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9f4337-6892-4adf-8420-a201728544c0 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:51.945481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273dd5d1-60af-4fdc-9949-b01228cfc22e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:51.945910 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273dd5d1-60af-4fdc-9949-b01228cfc22e admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:51.946193 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273dd5d1-60af-4fdc-9949-b01228cfc22e admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:51.946467 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273dd5d1-60af-4fdc-9949-b01228cfc22e admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:51.947108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273dd5d1-60af-4fdc-9949-b01228cfc22e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:04:51.947412 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273dd5d1-60af-4fdc-9949-b01228cfc22e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:51.948981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273dd5d1-60af-4fdc-9949-b01228cfc22e admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:51.953402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 796/1569] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:04:51 2022] POST /identity/v3/projects => generated 373 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:51.953402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171436f5-d382-4a2c-98ca-c2d457dfb579 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:51.964796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 774/1570] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:04:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:04:51.965635 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273dd5d1-60af-4fdc-9949-b01228cfc22e admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:51.975692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273dd5d1-60af-4fdc-9949-b01228cfc22e admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:52.018995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171436f5-d382-4a2c-98ca-c2d457dfb579 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:52.023554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064d4b49-db25-42a4-ac82-b7ccf70bd9a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:52.024149 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064d4b49-db25-42a4-ac82-b7ccf70bd9a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:52.024427 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064d4b49-db25-42a4-ac82-b7ccf70bd9a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:52.024692 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064d4b49-db25-42a4-ac82-b7ccf70bd9a7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:52.025361 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064d4b49-db25-42a4-ac82-b7ccf70bd9a7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:04:52.025664 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064d4b49-db25-42a4-ac82-b7ccf70bd9a7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:52.027255 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064d4b49-db25-42a4-ac82-b7ccf70bd9a7 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:52.032000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c401b442-9cdb-41b4-981c-ec462ca55ab7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:52.032605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c401b442-9cdb-41b4-981c-ec462ca55ab7 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:52.032886 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c401b442-9cdb-41b4-981c-ec462ca55ab7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:52.033190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c401b442-9cdb-41b4-981c-ec462ca55ab7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:52.033841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c401b442-9cdb-41b4-981c-ec462ca55ab7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:52.035445 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c401b442-9cdb-41b4-981c-ec462ca55ab7 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:52.039091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 797/1571] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:04:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:04:52.039873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064d4b49-db25-42a4-ac82-b7ccf70bd9a7 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:52.049143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064d4b49-db25-42a4-ac82-b7ccf70bd9a7 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:52.053921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 775/1572] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:51 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:52.054603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8472f6-b625-4735-bebe-64ec9afdf610 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:52.055384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8472f6-b625-4735-bebe-64ec9afdf610 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:52.055824 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8472f6-b625-4735-bebe-64ec9afdf610 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:52.056162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9a9242-a1cc-4bb8-8c2d-d4a1e6716b27 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:52.056305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8472f6-b625-4735-bebe-64ec9afdf610 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:52.056479 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9a9242-a1cc-4bb8-8c2d-d4a1e6716b27 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:52.056733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9a9242-a1cc-4bb8-8c2d-d4a1e6716b27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:52.056994 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8472f6-b625-4735-bebe-64ec9afdf610 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:52.059008 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8472f6-b625-4735-bebe-64ec9afdf610 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:52.075485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-2b9a9242-a1cc-4bb8-8c2d-d4a1e6716b27 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:52.077265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2b9a9242-a1cc-4bb8-8c2d-d4a1e6716b27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:52.084505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 776/1573] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:52.085311 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9a9242-a1cc-4bb8-8c2d-d4a1e6716b27 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:52.094758 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9a9242-a1cc-4bb8-8c2d-d4a1e6716b27 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:52.100141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0acaa28-59e8-4b23-88bb-8dd82c322185 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:52.103659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0acaa28-59e8-4b23-88bb-8dd82c322185 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:52.103659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0acaa28-59e8-4b23-88bb-8dd82c322185 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:52.103659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0acaa28-59e8-4b23-88bb-8dd82c322185 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:52.103659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0acaa28-59e8-4b23-88bb-8dd82c322185 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:04:52.103659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0acaa28-59e8-4b23-88bb-8dd82c322185 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:52.104141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0acaa28-59e8-4b23-88bb-8dd82c322185 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:52.116281 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 777/1574] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:04:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:04:52.117035 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0acaa28-59e8-4b23-88bb-8dd82c322185 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:52.126325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0acaa28-59e8-4b23-88bb-8dd82c322185 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:52.131483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a596752-1184-42d5-97d5-a5dab3a17ebd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:52.137161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a596752-1184-42d5-97d5-a5dab3a17ebd admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:52.137161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a596752-1184-42d5-97d5-a5dab3a17ebd admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:52.137161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a596752-1184-42d5-97d5-a5dab3a17ebd admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:52.137161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a596752-1184-42d5-97d5-a5dab3a17ebd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:52.137161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a596752-1184-42d5-97d5-a5dab3a17ebd admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:52.160741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 798/1575] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:04:52 2022] POST /identity/v3/projects => generated 389 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:52.165851 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8472f6-b625-4735-bebe-64ec9afdf610 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:52.167655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:52.223049 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8472f6-b625-4735-bebe-64ec9afdf610 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:52.228633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6073cbeb-98ca-4f05-bb78-4465b35f57f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:52.229277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6073cbeb-98ca-4f05-bb78-4465b35f57f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:52.229570 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6073cbeb-98ca-4f05-bb78-4465b35f57f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:52.229844 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6073cbeb-98ca-4f05-bb78-4465b35f57f9 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:52.230070 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 778/1576] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:04:52 2022] POST /identity/v3/projects => generated 377 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:52.230623 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6073cbeb-98ca-4f05-bb78-4465b35f57f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:52.231176 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a596752-1184-42d5-97d5-a5dab3a17ebd admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:52.231659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: WARNING py.warnings [None req-6073cbeb-98ca-4f05-bb78-4465b35f57f9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 04 01:04:52.231659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: warnings.warn(msg) Sep 04 01:04:52.231659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: Sep 04 01:04:52.232965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6073cbeb-98ca-4f05-bb78-4465b35f57f9 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:52.246401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:52.294162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a596752-1184-42d5-97d5-a5dab3a17ebd admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:52.298897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6057c995-02b1-4014-abd3-21bc048b52c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:52.299497 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057c995-02b1-4014-abd3-21bc048b52c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:52.299777 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057c995-02b1-4014-abd3-21bc048b52c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:52.300093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057c995-02b1-4014-abd3-21bc048b52c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:52.300718 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6057c995-02b1-4014-abd3-21bc048b52c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:52.302687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6057c995-02b1-4014-abd3-21bc048b52c6 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:52.347968 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 799/1577] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:04:52 2022] POST /identity/v3/users => generated 287 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:52.348922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6073cbeb-98ca-4f05-bb78-4465b35f57f9 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:52.359263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6073cbeb-98ca-4f05-bb78-4465b35f57f9 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:52.364134 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65e3bbe-c533-4a33-89d1-f98232a4451f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:52.364717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65e3bbe-c533-4a33-89d1-f98232a4451f admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:52.365196 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65e3bbe-c533-4a33-89d1-f98232a4451f admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:52.365627 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65e3bbe-c533-4a33-89d1-f98232a4451f admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:52.366251 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65e3bbe-c533-4a33-89d1-f98232a4451f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:52.368231 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65e3bbe-c533-4a33-89d1-f98232a4451f admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:52.436459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 779/1578] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:04:52 2022] POST /identity/v3/projects => generated 393 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:52.437332 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6057c995-02b1-4014-abd3-21bc048b52c6 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:52.464263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 800/1579] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:04:52 2022] POST /identity/v3/projects => generated 383 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:52.466424 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65e3bbe-c533-4a33-89d1-f98232a4451f admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:52.468997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d65e3bbe-c533-4a33-89d1-f98232a4451f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:52.503469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6057c995-02b1-4014-abd3-21bc048b52c6 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:52.526956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65e3bbe-c533-4a33-89d1-f98232a4451f admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:52.531019 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904b05c4-7b8d-4848-a73e-4c112b3d3197 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:52.531633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904b05c4-7b8d-4848-a73e-4c112b3d3197 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:52.531913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904b05c4-7b8d-4848-a73e-4c112b3d3197 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:52.532195 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904b05c4-7b8d-4848-a73e-4c112b3d3197 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:52.562480 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569ae416-cc4a-4acd-a334-bf5a504c664e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:52.563481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569ae416-cc4a-4acd-a334-bf5a504c664e admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:52.563718 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569ae416-cc4a-4acd-a334-bf5a504c664e admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:52.564001 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569ae416-cc4a-4acd-a334-bf5a504c664e admin admin] PATH_INFO: `/v3/projects/cd838d15487d45ae99751bc557b4a742/users/28a060c037f1473d99a9857496ceb116/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:52.578017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569ae416-cc4a-4acd-a334-bf5a504c664e admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd838d15487d45ae99751bc557b4a742, user_id=28a060c037f1473d99a9857496ceb116, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:52.580729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569ae416-cc4a-4acd-a334-bf5a504c664e admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:52.595684 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904b05c4-7b8d-4848-a73e-4c112b3d3197 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:52.597158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904b05c4-7b8d-4848-a73e-4c112b3d3197 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:52.609554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 801/1580] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:04:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:04:52.610489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904b05c4-7b8d-4848-a73e-4c112b3d3197 service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:52.622693 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904b05c4-7b8d-4848-a73e-4c112b3d3197 service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:52.627669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 780/1581] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:04:52 2022] PUT /identity/v3/projects/cd838d15487d45ae99751bc557b4a742/users/28a060c037f1473d99a9857496ceb116/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:04:52.628349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569ae416-cc4a-4acd-a334-bf5a504c664e admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:52.648637 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:52.667169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:52.692719 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c7ae3d-c3ba-43eb-936b-82bb1c76e932 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:52.695854 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c7ae3d-c3ba-43eb-936b-82bb1c76e932 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:52.695854 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c7ae3d-c3ba-43eb-936b-82bb1c76e932 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:52.695854 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c7ae3d-c3ba-43eb-936b-82bb1c76e932 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:52.695854 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c7ae3d-c3ba-43eb-936b-82bb1c76e932 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:52.697132 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c7ae3d-c3ba-43eb-936b-82bb1c76e932 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:52.700232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569ae416-cc4a-4acd-a334-bf5a504c664e admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:52.705030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808dca0a-3d66-4636-9312-6e7eaa382512 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:52.705710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808dca0a-3d66-4636-9312-6e7eaa382512 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:52.706091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808dca0a-3d66-4636-9312-6e7eaa382512 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:52.706326 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808dca0a-3d66-4636-9312-6e7eaa382512 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:52.706888 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808dca0a-3d66-4636-9312-6e7eaa382512 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:52.709834 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808dca0a-3d66-4636-9312-6e7eaa382512 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:52.821239 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 802/1582] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:04:52 2022] POST /identity/v3/projects => generated 405 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:52.821239 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c7ae3d-c3ba-43eb-936b-82bb1c76e932 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:52.851158 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 781/1583] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:04:52 2022] POST /identity/v3/users => generated 295 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:52.851998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808dca0a-3d66-4636-9312-6e7eaa382512 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:52.861393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808dca0a-3d66-4636-9312-6e7eaa382512 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:52.883172 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c7ae3d-c3ba-43eb-936b-82bb1c76e932 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:52.887965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96a6937-4ff6-459d-9e6d-9aa8db0ba36d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:52.888811 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96a6937-4ff6-459d-9e6d-9aa8db0ba36d admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:52.889125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96a6937-4ff6-459d-9e6d-9aa8db0ba36d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:52.889386 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96a6937-4ff6-459d-9e6d-9aa8db0ba36d admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:52.896322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96a6937-4ff6-459d-9e6d-9aa8db0ba36d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:52.896322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96a6937-4ff6-459d-9e6d-9aa8db0ba36d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:52.923035 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a92dfc-c56d-4309-b262-b4fd299a3d57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:52.923035 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a92dfc-c56d-4309-b262-b4fd299a3d57 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:52.923035 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a92dfc-c56d-4309-b262-b4fd299a3d57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:52.923035 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a92dfc-c56d-4309-b262-b4fd299a3d57 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:52.923460 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a92dfc-c56d-4309-b262-b4fd299a3d57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:52.924260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a92dfc-c56d-4309-b262-b4fd299a3d57 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:52.938041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 782/1584] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:52 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:52.938041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a92dfc-c56d-4309-b262-b4fd299a3d57 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:52.948632 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a92dfc-c56d-4309-b262-b4fd299a3d57 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:52.962026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dff28d-59d5-474d-90b7-eea8800a6842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:52.962644 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dff28d-59d5-474d-90b7-eea8800a6842 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:52.962925 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dff28d-59d5-474d-90b7-eea8800a6842 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:52.963196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dff28d-59d5-474d-90b7-eea8800a6842 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:52.963834 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dff28d-59d5-474d-90b7-eea8800a6842 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:52.965965 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dff28d-59d5-474d-90b7-eea8800a6842 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.022685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 803/1585] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:04:52 2022] POST /identity/v3/users => generated 289 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:53.023737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96a6937-4ff6-459d-9e6d-9aa8db0ba36d admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.033501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96a6937-4ff6-459d-9e6d-9aa8db0ba36d admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.039104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df70b5df-33d5-48a0-b0fb-f4efc55c2b57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.039708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df70b5df-33d5-48a0-b0fb-f4efc55c2b57 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.040014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df70b5df-33d5-48a0-b0fb-f4efc55c2b57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.040286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df70b5df-33d5-48a0-b0fb-f4efc55c2b57 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.040913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df70b5df-33d5-48a0-b0fb-f4efc55c2b57 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.042928 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df70b5df-33d5-48a0-b0fb-f4efc55c2b57 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.054067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 783/1586] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:04:52 2022] POST /identity/v3/users => generated 297 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:53.054067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbc65e7-c124-4180-9909-6073f9167d1a None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.054067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbc65e7-c124-4180-9909-6073f9167d1a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.054067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbc65e7-c124-4180-9909-6073f9167d1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.084255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-cbbc65e7-c124-4180-9909-6073f9167d1a None None] MFA Rules not processed for user `28a060c037f1473d99a9857496ceb116`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:53.085518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-cbbc65e7-c124-4180-9909-6073f9167d1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:53.151720 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 784/1587] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:53 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:53.152963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbc65e7-c124-4180-9909-6073f9167d1a None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.156133 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 804/1588] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:04:53 2022] POST /identity/v3/users => generated 292 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:53.157249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df70b5df-33d5-48a0-b0fb-f4efc55c2b57 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.168336 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbc65e7-c124-4180-9909-6073f9167d1a None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.170218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df70b5df-33d5-48a0-b0fb-f4efc55c2b57 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.172912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f090f859-7fc5-4698-8322-37ec2a1b6d09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.173555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f090f859-7fc5-4698-8322-37ec2a1b6d09 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.173838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f090f859-7fc5-4698-8322-37ec2a1b6d09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.174104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f090f859-7fc5-4698-8322-37ec2a1b6d09 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.174772 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f090f859-7fc5-4698-8322-37ec2a1b6d09 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.175581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b28dcd-b10f-4753-b61a-f301fedcf1b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.176302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b28dcd-b10f-4753-b61a-f301fedcf1b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.176590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b28dcd-b10f-4753-b61a-f301fedcf1b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.176704 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f090f859-7fc5-4698-8322-37ec2a1b6d09 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.176896 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b28dcd-b10f-4753-b61a-f301fedcf1b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.177556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b28dcd-b10f-4753-b61a-f301fedcf1b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.179294 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b28dcd-b10f-4753-b61a-f301fedcf1b8 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.200353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 805/1589] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:53 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:53.201251 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b28dcd-b10f-4753-b61a-f301fedcf1b8 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.224902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b28dcd-b10f-4753-b61a-f301fedcf1b8 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.230268 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b688e57-65c0-4643-aa35-960b0411ca66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.230880 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b688e57-65c0-4643-aa35-960b0411ca66 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.231292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b688e57-65c0-4643-aa35-960b0411ca66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.231566 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b688e57-65c0-4643-aa35-960b0411ca66 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.232347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b688e57-65c0-4643-aa35-960b0411ca66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.233977 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b688e57-65c0-4643-aa35-960b0411ca66 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.245438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 806/1590] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:53 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:53.246326 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b688e57-65c0-4643-aa35-960b0411ca66 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.256995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b688e57-65c0-4643-aa35-960b0411ca66 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.261627 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4683a2f3-feea-420a-b77a-7f9bc4f25f85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.262213 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4683a2f3-feea-420a-b77a-7f9bc4f25f85 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.262516 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4683a2f3-feea-420a-b77a-7f9bc4f25f85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.262781 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4683a2f3-feea-420a-b77a-7f9bc4f25f85 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.263413 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4683a2f3-feea-420a-b77a-7f9bc4f25f85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.264994 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4683a2f3-feea-420a-b77a-7f9bc4f25f85 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.275497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 807/1591] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:53 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:53.276493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4683a2f3-feea-420a-b77a-7f9bc4f25f85 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.286453 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4683a2f3-feea-420a-b77a-7f9bc4f25f85 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.291539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5710fcad-b26d-4552-9211-a6293dc7f223 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.297222 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5710fcad-b26d-4552-9211-a6293dc7f223 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.297222 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5710fcad-b26d-4552-9211-a6293dc7f223 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.297222 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5710fcad-b26d-4552-9211-a6293dc7f223 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.297488 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 785/1592] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:04:53 2022] POST /identity/v3/users => generated 303 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:53.297488 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f090f859-7fc5-4698-8322-37ec2a1b6d09 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.297666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5710fcad-b26d-4552-9211-a6293dc7f223 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.297666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5710fcad-b26d-4552-9211-a6293dc7f223 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.303855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f090f859-7fc5-4698-8322-37ec2a1b6d09 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.308091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 808/1593] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:53 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:53.308757 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5710fcad-b26d-4552-9211-a6293dc7f223 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.311744 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2087bb0a-6d43-4da1-9a16-4264f9d41c87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.312351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2087bb0a-6d43-4da1-9a16-4264f9d41c87 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.312627 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2087bb0a-6d43-4da1-9a16-4264f9d41c87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.312897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2087bb0a-6d43-4da1-9a16-4264f9d41c87 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.313559 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2087bb0a-6d43-4da1-9a16-4264f9d41c87 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.315519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2087bb0a-6d43-4da1-9a16-4264f9d41c87 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.320185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5710fcad-b26d-4552-9211-a6293dc7f223 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.322450 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=91,keystone:INSERT=9 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:53.325243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96784eef-daa1-4dac-8565-f9c8bb37dc6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.325914 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96784eef-daa1-4dac-8565-f9c8bb37dc6d admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.326197 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96784eef-daa1-4dac-8565-f9c8bb37dc6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.326472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96784eef-daa1-4dac-8565-f9c8bb37dc6d admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.327219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96784eef-daa1-4dac-8565-f9c8bb37dc6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.328869 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96784eef-daa1-4dac-8565-f9c8bb37dc6d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.342496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 809/1594] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:53 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:53.342496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96784eef-daa1-4dac-8565-f9c8bb37dc6d admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.353776 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96784eef-daa1-4dac-8565-f9c8bb37dc6d admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.358573 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b443e5-ec6f-4b01-9590-2ca37ef1630d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.359190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b443e5-ec6f-4b01-9590-2ca37ef1630d admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.359471 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b443e5-ec6f-4b01-9590-2ca37ef1630d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.359744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b443e5-ec6f-4b01-9590-2ca37ef1630d admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.360381 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b443e5-ec6f-4b01-9590-2ca37ef1630d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.362342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b443e5-ec6f-4b01-9590-2ca37ef1630d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.373035 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 810/1595] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:53 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:53.373852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b443e5-ec6f-4b01-9590-2ca37ef1630d admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.384362 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b443e5-ec6f-4b01-9590-2ca37ef1630d admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.389602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7fd03a-2e7e-44d1-bafd-a992a5d73ba8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.390275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7fd03a-2e7e-44d1-bafd-a992a5d73ba8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.390554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7fd03a-2e7e-44d1-bafd-a992a5d73ba8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.390826 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7fd03a-2e7e-44d1-bafd-a992a5d73ba8 admin admin] PATH_INFO: `/v3/projects/959a493c8f8e49c990227d2229946a4f/users/2fb029a637c74b47a227abf80b968af9/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.403377 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:53.407267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 786/1596] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:04:53 2022] POST /identity/v3/projects => generated 385 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:53.408200 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7fd03a-2e7e-44d1-bafd-a992a5d73ba8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=959a493c8f8e49c990227d2229946a4f, user_id=2fb029a637c74b47a227abf80b968af9, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.408355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2087bb0a-6d43-4da1-9a16-4264f9d41c87 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.410797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7fd03a-2e7e-44d1-bafd-a992a5d73ba8 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.471849 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 811/1597] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:04:53 2022] PUT /identity/v3/projects/959a493c8f8e49c990227d2229946a4f/users/2fb029a637c74b47a227abf80b968af9/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:04:53.481307 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7fd03a-2e7e-44d1-bafd-a992a5d73ba8 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.484444 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2087bb0a-6d43-4da1-9a16-4264f9d41c87 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.542125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7fd03a-2e7e-44d1-bafd-a992a5d73ba8 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.543314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83a04f9-39f3-40b6-8370-e72703f78bff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.543928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83a04f9-39f3-40b6-8370-e72703f78bff admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.544210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83a04f9-39f3-40b6-8370-e72703f78bff admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.544540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83a04f9-39f3-40b6-8370-e72703f78bff admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.545231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83a04f9-39f3-40b6-8370-e72703f78bff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.546832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83a04f9-39f3-40b6-8370-e72703f78bff admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.547383 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5868ab9-74ff-4700-85fd-c62416f7eaa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.548072 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5868ab9-74ff-4700-85fd-c62416f7eaa1 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.548361 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5868ab9-74ff-4700-85fd-c62416f7eaa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.548631 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5868ab9-74ff-4700-85fd-c62416f7eaa1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.549313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5868ab9-74ff-4700-85fd-c62416f7eaa1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.550939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5868ab9-74ff-4700-85fd-c62416f7eaa1 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.560565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 787/1598] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:53 2022] GET /identity/v3/roles => generated 1420 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:53.561811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83a04f9-39f3-40b6-8370-e72703f78bff admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.564728 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 812/1599] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:53 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:53.565590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5868ab9-74ff-4700-85fd-c62416f7eaa1 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.573993 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83a04f9-39f3-40b6-8370-e72703f78bff admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.575520 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5868ab9-74ff-4700-85fd-c62416f7eaa1 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.579157 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249abc2a-87ca-40cb-a4a0-598545ff5312 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.579892 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249abc2a-87ca-40cb-a4a0-598545ff5312 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.580180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249abc2a-87ca-40cb-a4a0-598545ff5312 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.580374 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fc1d79-1207-4f2a-a12c-e9ad6dfa84a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.580513 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249abc2a-87ca-40cb-a4a0-598545ff5312 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.581438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fc1d79-1207-4f2a-a12c-e9ad6dfa84a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.581552 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249abc2a-87ca-40cb-a4a0-598545ff5312 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.581631 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fc1d79-1207-4f2a-a12c-e9ad6dfa84a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.581631 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fc1d79-1207-4f2a-a12c-e9ad6dfa84a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.582139 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65fc1d79-1207-4f2a-a12c-e9ad6dfa84a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.582711 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249abc2a-87ca-40cb-a4a0-598545ff5312 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.583865 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65fc1d79-1207-4f2a-a12c-e9ad6dfa84a9 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.594555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 788/1600] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:53 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:53.594864 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249abc2a-87ca-40cb-a4a0-598545ff5312 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.602326 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 813/1601] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:53 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:53.603450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fc1d79-1207-4f2a-a12c-e9ad6dfa84a9 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.606481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249abc2a-87ca-40cb-a4a0-598545ff5312 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.612112 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427d36e2-39ab-4134-a602-3d06395d3e84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.612826 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427d36e2-39ab-4134-a602-3d06395d3e84 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.613142 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427d36e2-39ab-4134-a602-3d06395d3e84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.613422 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427d36e2-39ab-4134-a602-3d06395d3e84 admin admin] PATH_INFO: `/v3/projects/55faed42b7f243df9df879d581797f56/users/fbcd4c19e766446188e3b3f41a44e2e4/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.614650 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fc1d79-1207-4f2a-a12c-e9ad6dfa84a9 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.621830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec48ad0-4690-4d29-ac4b-e341155f1285 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.622596 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec48ad0-4690-4d29-ac4b-e341155f1285 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.622961 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec48ad0-4690-4d29-ac4b-e341155f1285 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.623513 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec48ad0-4690-4d29-ac4b-e341155f1285 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.623940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec48ad0-4690-4d29-ac4b-e341155f1285 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.627551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec48ad0-4690-4d29-ac4b-e341155f1285 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.632506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427d36e2-39ab-4134-a602-3d06395d3e84 admin admin] RBAC: Authorizing `identity:create_grant(project_id=55faed42b7f243df9df879d581797f56, user_id=fbcd4c19e766446188e3b3f41a44e2e4, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.635444 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427d36e2-39ab-4134-a602-3d06395d3e84 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.694558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 789/1602] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:04:53 2022] PUT /identity/v3/projects/55faed42b7f243df9df879d581797f56/users/fbcd4c19e766446188e3b3f41a44e2e4/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:04:53.695804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bbcd92-6c4b-4c58-85ac-77496a2c310d None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.696087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bbcd92-6c4b-4c58-85ac-77496a2c310d None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.696347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bbcd92-6c4b-4c58-85ac-77496a2c310d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.718004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 814/1603] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:04:53 2022] POST /identity/v3/users => generated 299 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:53.718657 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec48ad0-4690-4d29-ac4b-e341155f1285 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.727305 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-47bbcd92-6c4b-4c58-85ac-77496a2c310d None None] MFA Rules not processed for user `2fb029a637c74b47a227abf80b968af9`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:53.728511 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-47bbcd92-6c4b-4c58-85ac-77496a2c310d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:53.729661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec48ad0-4690-4d29-ac4b-e341155f1285 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.774044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:53.793679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2167d764-d8f0-4869-85d9-679d67343562 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.794477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2167d764-d8f0-4869-85d9-679d67343562 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.795006 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2167d764-d8f0-4869-85d9-679d67343562 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.795150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2167d764-d8f0-4869-85d9-679d67343562 admin admin] PATH_INFO: `/v3/projects/e7dca68418a14d7da9aa63ac25f202e8/users/44993cd93360442bb0afb28778ee83bc/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.799226 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 790/1604] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:53 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:53.799868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bbcd92-6c4b-4c58-85ac-77496a2c310d None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.814630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bbcd92-6c4b-4c58-85ac-77496a2c310d None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.817295 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2167d764-d8f0-4869-85d9-679d67343562 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e7dca68418a14d7da9aa63ac25f202e8, user_id=44993cd93360442bb0afb28778ee83bc, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.820407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25300e85-7000-47c6-8b4d-c4f03400d94e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.820585 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2167d764-d8f0-4869-85d9-679d67343562 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.821190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25300e85-7000-47c6-8b4d-c4f03400d94e admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.821487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25300e85-7000-47c6-8b4d-c4f03400d94e admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.821761 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25300e85-7000-47c6-8b4d-c4f03400d94e admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.822565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25300e85-7000-47c6-8b4d-c4f03400d94e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.824177 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25300e85-7000-47c6-8b4d-c4f03400d94e admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.836046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 791/1605] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:53 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:53.836855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25300e85-7000-47c6-8b4d-c4f03400d94e admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.848676 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25300e85-7000-47c6-8b4d-c4f03400d94e admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:53.857775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861ad8ff-b33a-4daa-b877-a671c0eae7a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:53.858595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861ad8ff-b33a-4daa-b877-a671c0eae7a8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.858877 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861ad8ff-b33a-4daa-b877-a671c0eae7a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.859156 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861ad8ff-b33a-4daa-b877-a671c0eae7a8 admin admin] PATH_INFO: `/v3/projects/db72b63281384bd883b04d05bcc35677/users/bd13df3a2d0e4182bbd86dab7035c371/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.874985 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861ad8ff-b33a-4daa-b877-a671c0eae7a8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db72b63281384bd883b04d05bcc35677, user_id=bd13df3a2d0e4182bbd86dab7035c371, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:53.877792 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861ad8ff-b33a-4daa-b877-a671c0eae7a8 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:53.919819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 815/1606] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:04:53 2022] PUT /identity/v3/projects/e7dca68418a14d7da9aa63ac25f202e8/users/44993cd93360442bb0afb28778ee83bc/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:04:53.923975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2167d764-d8f0-4869-85d9-679d67343562 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:53.961265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 792/1607] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:04:53 2022] PUT /identity/v3/projects/db72b63281384bd883b04d05bcc35677/users/bd13df3a2d0e4182bbd86dab7035c371/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:04:53.964320 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73a7961-f84a-4018-badb-5741a2f0a81a None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:53.964864 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73a7961-f84a-4018-badb-5741a2f0a81a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:53.965612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73a7961-f84a-4018-badb-5741a2f0a81a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:53.970071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:53.986923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-c73a7961-f84a-4018-badb-5741a2f0a81a None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:53.988233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-c73a7961-f84a-4018-badb-5741a2f0a81a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:53.993513 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2167d764-d8f0-4869-85d9-679d67343562 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:54.054841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 793/1608] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:54.056067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73a7961-f84a-4018-badb-5741a2f0a81a None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:54.068608 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73a7961-f84a-4018-badb-5741a2f0a81a None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:54.068911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f581d1-0d08-4356-b8a0-d3065dd9c2d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:54.069710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f581d1-0d08-4356-b8a0-d3065dd9c2d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.069988 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f581d1-0d08-4356-b8a0-d3065dd9c2d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.070273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f581d1-0d08-4356-b8a0-d3065dd9c2d7 admin admin] PATH_INFO: `/v3/projects/287d111f08ac4b05bd5adec9c40d0620/users/f5e4723258c64ce5885154dded2d3d43/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.076739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5ab390-3f75-40ab-b7c2-7113d98b83ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:54.077806 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5ab390-3f75-40ab-b7c2-7113d98b83ee admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.078395 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5ab390-3f75-40ab-b7c2-7113d98b83ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.079407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5ab390-3f75-40ab-b7c2-7113d98b83ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.080525 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5ab390-3f75-40ab-b7c2-7113d98b83ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:54.082708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5ab390-3f75-40ab-b7c2-7113d98b83ee admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:54.089423 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f581d1-0d08-4356-b8a0-d3065dd9c2d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=287d111f08ac4b05bd5adec9c40d0620, user_id=f5e4723258c64ce5885154dded2d3d43, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:54.092173 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f581d1-0d08-4356-b8a0-d3065dd9c2d7 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:54.098390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 794/1609] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:54 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:54.103067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbf7fd8-c560-43fc-8e71-c047154959ee None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.103067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbf7fd8-c560-43fc-8e71-c047154959ee None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.103067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbf7fd8-c560-43fc-8e71-c047154959ee None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.104468 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 795/1610] 10.222.0.42 () {54 vars in 962 bytes} [Sun Sep 4 01:04:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:04:54.105475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbf7fd8-c560-43fc-8e71-c047154959ee None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:54.120823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbf7fd8-c560-43fc-8e71-c047154959ee None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:54.126239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cce777-7bb7-4375-a093-0ae0c32f1cc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:54.126901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cce777-7bb7-4375-a093-0ae0c32f1cc8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.127177 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cce777-7bb7-4375-a093-0ae0c32f1cc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.127447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cce777-7bb7-4375-a093-0ae0c32f1cc8 admin admin] PATH_INFO: `/v3/projects/8cd5d05953314eaab3d768e33897858f/users/5c092b8466ca48f8878491edb6dcb378/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.149629 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cce777-7bb7-4375-a093-0ae0c32f1cc8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8cd5d05953314eaab3d768e33897858f, user_id=5c092b8466ca48f8878491edb6dcb378, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:54.153749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cce777-7bb7-4375-a093-0ae0c32f1cc8 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:54.191649 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 816/1611] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:04:53 2022] PUT /identity/v3/projects/287d111f08ac4b05bd5adec9c40d0620/users/f5e4723258c64ce5885154dded2d3d43/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:04:54.195175 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60351c5-d3ee-4464-a600-fc9f62bb5429 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.195175 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60351c5-d3ee-4464-a600-fc9f62bb5429 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.195175 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60351c5-d3ee-4464-a600-fc9f62bb5429 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.214862 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-b60351c5-d3ee-4464-a600-fc9f62bb5429 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:54.216763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b60351c5-d3ee-4464-a600-fc9f62bb5429 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:54.220838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 796/1612] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:04:54 2022] PUT /identity/v3/projects/8cd5d05953314eaab3d768e33897858f/users/5c092b8466ca48f8878491edb6dcb378/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:04:54.222294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c57e8fd-b726-47b5-b055-ce23e12a29b0 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.226592 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c57e8fd-b726-47b5-b055-ce23e12a29b0 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.226592 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c57e8fd-b726-47b5-b055-ce23e12a29b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.242632 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-4c57e8fd-b726-47b5-b055-ce23e12a29b0 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:54.243905 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-4c57e8fd-b726-47b5-b055-ce23e12a29b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:54.289937 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 817/1613] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:54.289937 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60351c5-d3ee-4464-a600-fc9f62bb5429 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:54.312829 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 797/1614] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:54.313859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c57e8fd-b726-47b5-b055-ce23e12a29b0 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:54.330873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c57e8fd-b726-47b5-b055-ce23e12a29b0 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:54.336735 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d33b9a-92e4-4ba6-81c6-896b9995bfa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:54.337437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d33b9a-92e4-4ba6-81c6-896b9995bfa0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.337722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d33b9a-92e4-4ba6-81c6-896b9995bfa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.337993 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d33b9a-92e4-4ba6-81c6-896b9995bfa0 admin admin] PATH_INFO: `/v3/projects/ee5bd28fea9e4ab49b23f0ececbe3e26/users/4204338c7de24eaca2d191e9da3ec02d/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.354857 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:54.358089 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d33b9a-92e4-4ba6-81c6-896b9995bfa0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee5bd28fea9e4ab49b23f0ececbe3e26, user_id=4204338c7de24eaca2d191e9da3ec02d, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:54.361196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d33b9a-92e4-4ba6-81c6-896b9995bfa0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:54.373487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b60351c5-d3ee-4464-a600-fc9f62bb5429 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:54.379670 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987e8a36-825f-43d3-8f0a-d7793aece112 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:54.380851 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987e8a36-825f-43d3-8f0a-d7793aece112 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.381338 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987e8a36-825f-43d3-8f0a-d7793aece112 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.381891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987e8a36-825f-43d3-8f0a-d7793aece112 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.393351 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987e8a36-825f-43d3-8f0a-d7793aece112 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:54.395903 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987e8a36-825f-43d3-8f0a-d7793aece112 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:54.412009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 818/1615] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:04:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:04:54.413851 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12accd5-3064-4a19-a108-99223d9fed12 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.414211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12accd5-3064-4a19-a108-99223d9fed12 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.414595 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12accd5-3064-4a19-a108-99223d9fed12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.436783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 798/1616] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:04:54 2022] PUT /identity/v3/projects/ee5bd28fea9e4ab49b23f0ececbe3e26/users/4204338c7de24eaca2d191e9da3ec02d/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:04:54.440991 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd096d7-b865-4023-b64b-82be7cce83c4 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.441385 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd096d7-b865-4023-b64b-82be7cce83c4 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.441740 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd096d7-b865-4023-b64b-82be7cce83c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.472176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:54.473550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-5bd096d7-b865-4023-b64b-82be7cce83c4 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:54.475027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-5bd096d7-b865-4023-b64b-82be7cce83c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:54.487716 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-b12accd5-3064-4a19-a108-99223d9fed12 None None] MFA Rules not processed for user `57c88f16c7df40adb917f49804fda21f`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:54.488938 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b12accd5-3064-4a19-a108-99223d9fed12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:54.545899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 799/1617] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:54.547961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a3e9de-d99e-4711-ae92-81cd0f68e3af None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.548607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a3e9de-d99e-4711-ae92-81cd0f68e3af None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.549251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a3e9de-d99e-4711-ae92-81cd0f68e3af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.565335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 819/1618] 10.222.0.42 () {58 vars in 1059 bytes} [Sun Sep 4 01:04:54 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:54.566356 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12accd5-3064-4a19-a108-99223d9fed12 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:54.585782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-33a3e9de-d99e-4711-ae92-81cd0f68e3af None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:54.587941 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-33a3e9de-d99e-4711-ae92-81cd0f68e3af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:54.598150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 800/1619] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:54.599137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a3e9de-d99e-4711-ae92-81cd0f68e3af None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:54.666578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12accd5-3064-4a19-a108-99223d9fed12 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:54.671765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b24116-9792-4f3e-bdd6-3b8e80effd92 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:54.673000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b24116-9792-4f3e-bdd6-3b8e80effd92 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.673564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b24116-9792-4f3e-bdd6-3b8e80effd92 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.685154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b24116-9792-4f3e-bdd6-3b8e80effd92 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.701043 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b24116-9792-4f3e-bdd6-3b8e80effd92 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:54.702530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a3e9de-d99e-4711-ae92-81cd0f68e3af None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:54.703087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b24116-9792-4f3e-bdd6-3b8e80effd92 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:54.709856 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0cb69b8-487b-47d2-a9a3-4de2354b9585 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:54.710494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cb69b8-487b-47d2-a9a3-4de2354b9585 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.710774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cb69b8-487b-47d2-a9a3-4de2354b9585 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.711045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0cb69b8-487b-47d2-a9a3-4de2354b9585 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.717175 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 820/1620] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:04:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:04:54.718544 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b24116-9792-4f3e-bdd6-3b8e80effd92 service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:54.722161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0cb69b8-487b-47d2-a9a3-4de2354b9585 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:54.723930 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0cb69b8-487b-47d2-a9a3-4de2354b9585 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:54.728790 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b24116-9792-4f3e-bdd6-3b8e80effd92 service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:54.734963 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ea2b5e-e93d-446f-a1af-ab05e9be38d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:54.736072 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ea2b5e-e93d-446f-a1af-ab05e9be38d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.736909 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ea2b5e-e93d-446f-a1af-ab05e9be38d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.737380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ea2b5e-e93d-446f-a1af-ab05e9be38d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.738234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ea2b5e-e93d-446f-a1af-ab05e9be38d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:54.740602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ea2b5e-e93d-446f-a1af-ab05e9be38d0 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:54.741262 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 801/1621] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:04:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:04:54.746705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0cb69b8-487b-47d2-a9a3-4de2354b9585 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:54.755085 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0cb69b8-487b-47d2-a9a3-4de2354b9585 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:54.758727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 821/1622] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:54 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:54.759681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ea2b5e-e93d-446f-a1af-ab05e9be38d0 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:54.766571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae0eab4-7aa9-4203-81f5-164a0cf8a450 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:54.767846 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae0eab4-7aa9-4203-81f5-164a0cf8a450 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.768805 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae0eab4-7aa9-4203-81f5-164a0cf8a450 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.769410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae0eab4-7aa9-4203-81f5-164a0cf8a450 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.779811 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ea2b5e-e93d-446f-a1af-ab05e9be38d0 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:54.782060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae0eab4-7aa9-4203-81f5-164a0cf8a450 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:54.784341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae0eab4-7aa9-4203-81f5-164a0cf8a450 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:54.804224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cb403f-103e-4b12-9aa0-b351ed81aad5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:54.804224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cb403f-103e-4b12-9aa0-b351ed81aad5 service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.804224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cb403f-103e-4b12-9aa0-b351ed81aad5 service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.804224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cb403f-103e-4b12-9aa0-b351ed81aad5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.810820 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 802/1623] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:04:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:04:54.811699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae0eab4-7aa9-4203-81f5-164a0cf8a450 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:54.832981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae0eab4-7aa9-4203-81f5-164a0cf8a450 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:54.839540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67737886-403b-4b81-bccc-9c8a2a4705be service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:54.840205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67737886-403b-4b81-bccc-9c8a2a4705be service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.840525 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67737886-403b-4b81-bccc-9c8a2a4705be service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.840822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67737886-403b-4b81-bccc-9c8a2a4705be service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.875819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67737886-403b-4b81-bccc-9c8a2a4705be service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:54.875819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67737886-403b-4b81-bccc-9c8a2a4705be service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:54.882793 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cb403f-103e-4b12-9aa0-b351ed81aad5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:54.885120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cb403f-103e-4b12-9aa0-b351ed81aad5 service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:54.898140 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 803/1624] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:04:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:04:54.899243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67737886-403b-4b81-bccc-9c8a2a4705be service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:54.905515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 822/1625] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:04:54 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:04:54.918216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67737886-403b-4b81-bccc-9c8a2a4705be service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:54.934079 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203fd2b9-de10-4ece-97bb-6e4c06636c96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:54.935532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203fd2b9-de10-4ece-97bb-6e4c06636c96 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:54.935532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203fd2b9-de10-4ece-97bb-6e4c06636c96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:54.935532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203fd2b9-de10-4ece-97bb-6e4c06636c96 admin admin] PATH_INFO: `/v3/projects/ee5bd28fea9e4ab49b23f0ececbe3e26/users/4204338c7de24eaca2d191e9da3ec02d/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:54.938154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203fd2b9-de10-4ece-97bb-6e4c06636c96 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee5bd28fea9e4ab49b23f0ececbe3e26, user_id=4204338c7de24eaca2d191e9da3ec02d, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:54.940847 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203fd2b9-de10-4ece-97bb-6e4c06636c96 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:55.057268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 804/1626] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:04:54 2022] PUT /identity/v3/projects/ee5bd28fea9e4ab49b23f0ececbe3e26/users/4204338c7de24eaca2d191e9da3ec02d/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:04:55.066625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0845db15-7fda-48c0-adab-4a29d432a816 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:55.066625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0845db15-7fda-48c0-adab-4a29d432a816 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:55.066625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0845db15-7fda-48c0-adab-4a29d432a816 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:55.066625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-0845db15-7fda-48c0-adab-4a29d432a816 None None] MFA Rules not processed for user `4204338c7de24eaca2d191e9da3ec02d`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:55.068923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0845db15-7fda-48c0-adab-4a29d432a816 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:55.082031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:55.223157 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 823/1627] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:54 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:04:55.226144 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203fd2b9-de10-4ece-97bb-6e4c06636c96 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:55.324617 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:55.329518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203fd2b9-de10-4ece-97bb-6e4c06636c96 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:55.334148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3df601-5541-42bb-a972-056b40d8ee33 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:55.335910 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3df601-5541-42bb-a972-056b40d8ee33 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:55.336408 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3df601-5541-42bb-a972-056b40d8ee33 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:55.336989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3df601-5541-42bb-a972-056b40d8ee33 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:55.358903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3df601-5541-42bb-a972-056b40d8ee33 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:55.366498 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3df601-5541-42bb-a972-056b40d8ee33 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:55.402858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 805/1628] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:04:55 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:04:55.641248 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0845db15-7fda-48c0-adab-4a29d432a816 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:55.697677 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80de50a6-2a4d-4856-854a-4c9c08a0abf5 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:55.701955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80de50a6-2a4d-4856-854a-4c9c08a0abf5 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:55.701955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80de50a6-2a4d-4856-854a-4c9c08a0abf5 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:55.701955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 806/1629] 10.222.0.42 () {54 vars in 962 bytes} [Sun Sep 4 01:04:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 01:04:55.708228 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70902ed-6804-4574-b6b6-aee29fc6877b None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:55.708228 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70902ed-6804-4574-b6b6-aee29fc6877b None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:55.709557 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70902ed-6804-4574-b6b6-aee29fc6877b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:55.740171 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-e70902ed-6804-4574-b6b6-aee29fc6877b None None] MFA Rules not processed for user `57c88f16c7df40adb917f49804fda21f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:55.742516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e70902ed-6804-4574-b6b6-aee29fc6877b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:55.795420 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0845db15-7fda-48c0-adab-4a29d432a816 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:55.813958 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e871ffa7-1f97-47c3-bd6a-7b771e4d70dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:55.821529 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e871ffa7-1f97-47c3-bd6a-7b771e4d70dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:55.821529 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e871ffa7-1f97-47c3-bd6a-7b771e4d70dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:55.821529 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e871ffa7-1f97-47c3-bd6a-7b771e4d70dd admin admin] PATH_INFO: `/v3/users/28a060c037f1473d99a9857496ceb116` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:55.823677 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e871ffa7-1f97-47c3-bd6a-7b771e4d70dd admin admin] RBAC: Authorizing `identity:delete_user(user_id=28a060c037f1473d99a9857496ceb116)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:55.826408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e871ffa7-1f97-47c3-bd6a-7b771e4d70dd admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:55.881529 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 807/1630] 10.222.0.42 () {58 vars in 1059 bytes} [Sun Sep 4 01:04:55 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:04:55.906404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b153b174-f62b-4e82-ad99-11366f540741 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:55.907100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b153b174-f62b-4e82-ad99-11366f540741 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:55.908437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b153b174-f62b-4e82-ad99-11366f540741 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:55.912267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 808/1631] 10.222.0.42 () {54 vars in 912 bytes} [Sun Sep 4 01:04:55 2022] GET /identity => generated 270 bytes in 19 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:04:55.917108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7dbe980-5587-4e27-b67e-37e2546f3c1d None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:55.918718 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7dbe980-5587-4e27-b67e-37e2546f3c1d None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:55.919984 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7dbe980-5587-4e27-b67e-37e2546f3c1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:55.965622 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-d7dbe980-5587-4e27-b67e-37e2546f3c1d None None] MFA Rules not processed for user `57c88f16c7df40adb917f49804fda21f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:55.976205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d7dbe980-5587-4e27-b67e-37e2546f3c1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:55.993489 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 809/1632] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 4 01:04:55 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:56.008748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7dbe980-5587-4e27-b67e-37e2546f3c1d None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:56.022623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7dbe980-5587-4e27-b67e-37e2546f3c1d None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:56.027141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1036e261-771a-4074-bd92-b6a3e882a141 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:56.027881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1036e261-771a-4074-bd92-b6a3e882a141 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:56.028097 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1036e261-771a-4074-bd92-b6a3e882a141 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:56.028362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1036e261-771a-4074-bd92-b6a3e882a141 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:56.043289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1036e261-771a-4074-bd92-b6a3e882a141 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:56.044741 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1036e261-771a-4074-bd92-b6a3e882a141 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:56.451130 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 810/1633] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:04:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:04:56.462172 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e871ffa7-1f97-47c3-bd6a-7b771e4d70dd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28a060c037f1473d99a9857496ceb116'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:56.465978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23401b1-ab28-4fb0-9e7b-5a6985c55f99 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:56.465978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23401b1-ab28-4fb0-9e7b-5a6985c55f99 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:56.465978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23401b1-ab28-4fb0-9e7b-5a6985c55f99 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:56.465978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 811/1634] 10.222.0.42 () {54 vars in 912 bytes} [Sun Sep 4 01:04:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 01:04:56.465978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb91c698-1b0b-4105-807b-c0c2584d987e None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:56.465978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb91c698-1b0b-4105-807b-c0c2584d987e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:56.473940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb91c698-1b0b-4105-807b-c0c2584d987e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:56.473940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-fb91c698-1b0b-4105-807b-c0c2584d987e None None] MFA Rules not processed for user `57c88f16c7df40adb917f49804fda21f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:56.473940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-fb91c698-1b0b-4105-807b-c0c2584d987e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:56.473940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 812/1635] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 4 01:04:56 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:04:56.473940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb91c698-1b0b-4105-807b-c0c2584d987e None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:56.476637 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e871ffa7-1f97-47c3-bd6a-7b771e4d70dd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28a060c037f1473d99a9857496ceb116'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:56.479022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb91c698-1b0b-4105-807b-c0c2584d987e None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:56.479022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9430d2-e47d-41bf-9393-d02c11ca39f5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:56.479022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9430d2-e47d-41bf-9393-d02c11ca39f5 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:56.479022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9430d2-e47d-41bf-9393-d02c11ca39f5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:56.480732 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9430d2-e47d-41bf-9393-d02c11ca39f5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:56.481375 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e871ffa7-1f97-47c3-bd6a-7b771e4d70dd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28a060c037f1473d99a9857496ceb116'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:56.481375 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e871ffa7-1f97-47c3-bd6a-7b771e4d70dd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28a060c037f1473d99a9857496ceb116'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:56.483458 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9430d2-e47d-41bf-9393-d02c11ca39f5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:56.483458 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9430d2-e47d-41bf-9393-d02c11ca39f5 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:56.484556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e871ffa7-1f97-47c3-bd6a-7b771e4d70dd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28a060c037f1473d99a9857496ceb116'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:56.484556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 824/1636] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:04:55 2022] DELETE /identity/v3/users/28a060c037f1473d99a9857496ceb116 => generated 0 bytes in 767 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:04:56.484556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a228d9b5-c1c5-42a9-984c-3f227dff576a None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:56.484556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a228d9b5-c1c5-42a9-984c-3f227dff576a None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:56.484556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a228d9b5-c1c5-42a9-984c-3f227dff576a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:56.485135 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 813/1637] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:04:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:04:56.495985 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-a228d9b5-c1c5-42a9-984c-3f227dff576a None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:56.500343 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a228d9b5-c1c5-42a9-984c-3f227dff576a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:56.584330 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:56.624501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9430d2-e47d-41bf-9393-d02c11ca39f5 service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:56.645658 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9430d2-e47d-41bf-9393-d02c11ca39f5 service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:56.705193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 825/1638] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:04:56.756070 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1626d1e-7329-4a89-87c1-c0f93d9eb017 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:56.757100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1626d1e-7329-4a89-87c1-c0f93d9eb017 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:56.758018 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1626d1e-7329-4a89-87c1-c0f93d9eb017 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:56.758420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1626d1e-7329-4a89-87c1-c0f93d9eb017 admin admin] PATH_INFO: `/v3/users/2fb029a637c74b47a227abf80b968af9` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:56.759897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1626d1e-7329-4a89-87c1-c0f93d9eb017 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2fb029a637c74b47a227abf80b968af9)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:56.761434 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: WARNING py.warnings [None req-c1626d1e-7329-4a89-87c1-c0f93d9eb017 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 04 01:04:56.761434 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: warnings.warn(msg) Sep 04 01:04:56.761434 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: Sep 04 01:04:56.766997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a228d9b5-c1c5-42a9-984c-3f227dff576a None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:56.769217 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a228d9b5-c1c5-42a9-984c-3f227dff576a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:56.776066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1626d1e-7329-4a89-87c1-c0f93d9eb017 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:56.896376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a228d9b5-c1c5-42a9-984c-3f227dff576a None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:56.910895 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41838cfb-3730-4ce1-8083-2e0e40eb5d91 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:56.912415 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41838cfb-3730-4ce1-8083-2e0e40eb5d91 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:56.913288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41838cfb-3730-4ce1-8083-2e0e40eb5d91 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:56.914232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41838cfb-3730-4ce1-8083-2e0e40eb5d91 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:56.930452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41838cfb-3730-4ce1-8083-2e0e40eb5d91 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:56.932229 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41838cfb-3730-4ce1-8083-2e0e40eb5d91 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:56.946772 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 826/1639] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:04:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:04:57.139608 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:57.147573 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c1626d1e-7329-4a89-87c1-c0f93d9eb017 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2fb029a637c74b47a227abf80b968af9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:57.149044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c1626d1e-7329-4a89-87c1-c0f93d9eb017 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2fb029a637c74b47a227abf80b968af9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:57.290949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c1626d1e-7329-4a89-87c1-c0f93d9eb017 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2fb029a637c74b47a227abf80b968af9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:57.326652 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c1626d1e-7329-4a89-87c1-c0f93d9eb017 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2fb029a637c74b47a227abf80b968af9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:57.328459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c1626d1e-7329-4a89-87c1-c0f93d9eb017 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2fb029a637c74b47a227abf80b968af9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:57.359252 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 814/1640] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:04:56 2022] DELETE /identity/v3/users/2fb029a637c74b47a227abf80b968af9 => generated 0 bytes in 735 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:04:57.363995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da246186-595c-47ba-9e99-2dff37bc0163 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:57.365390 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da246186-595c-47ba-9e99-2dff37bc0163 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:57.365889 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da246186-595c-47ba-9e99-2dff37bc0163 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:57.398769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-da246186-595c-47ba-9e99-2dff37bc0163 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:57.401062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-da246186-595c-47ba-9e99-2dff37bc0163 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:57.525116 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 827/1641] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:04:57.546592 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1626d1e-7329-4a89-87c1-c0f93d9eb017 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:57.548151 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-c1626d1e-7329-4a89-87c1-c0f93d9eb017 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:57.653394 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1626d1e-7329-4a89-87c1-c0f93d9eb017 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:57.658078 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8723b1c1-ea22-4318-9e45-c8d38434cb4b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:57.658757 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8723b1c1-ea22-4318-9e45-c8d38434cb4b service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:57.660802 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8723b1c1-ea22-4318-9e45-c8d38434cb4b service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:57.660802 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8723b1c1-ea22-4318-9e45-c8d38434cb4b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:57.671462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8723b1c1-ea22-4318-9e45-c8d38434cb4b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:57.673526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8723b1c1-ea22-4318-9e45-c8d38434cb4b service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:57.686132 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 815/1642] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:04:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:04:58.092603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da246186-595c-47ba-9e99-2dff37bc0163 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:58.094209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-da246186-595c-47ba-9e99-2dff37bc0163 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:58.117268 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da246186-595c-47ba-9e99-2dff37bc0163 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:58.128700 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:58.129529 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:58.130571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:58.130733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e admin admin] PATH_INFO: `/v3/projects/cd838d15487d45ae99751bc557b4a742` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:58.131954 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e admin admin] RBAC: Authorizing `identity:delete_project(project_id=cd838d15487d45ae99751bc557b4a742)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:58.133089 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: WARNING py.warnings [None req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 04 01:04:58.133089 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: warnings.warn(msg) Sep 04 01:04:58.133089 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: Sep 04 01:04:58.138325 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:58.178454 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5db96f-e0f4-438a-b875-2f2a29f39ec9 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:58.180841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5db96f-e0f4-438a-b875-2f2a29f39ec9 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:58.181418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5db96f-e0f4-438a-b875-2f2a29f39ec9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:58.232008 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-7d5db96f-e0f4-438a-b875-2f2a29f39ec9 None None] MFA Rules not processed for user `5c092b8466ca48f8878491edb6dcb378`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:58.238158 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-7d5db96f-e0f4-438a-b875-2f2a29f39ec9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:58.333168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cd838d15487d45ae99751bc557b4a742', 'request_id': 'req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:58.352208 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cd838d15487d45ae99751bc557b4a742', 'request_id': 'req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:58.392155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cd838d15487d45ae99751bc557b4a742', 'request_id': 'req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:58.399104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:58.406825 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 816/1643] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:58 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:04:58.408981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5db96f-e0f4-438a-b875-2f2a29f39ec9 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:58.411728 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-7d5db96f-e0f4-438a-b875-2f2a29f39ec9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:58.457224 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5db96f-e0f4-438a-b875-2f2a29f39ec9 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:58.489161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e admin admin] The token cache is being invalidate because project cd838d15487d45ae99751bc557b4a742 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:04:58.491579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:58.493452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:58.496983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 828/1644] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:04:58 2022] DELETE /identity/v3/projects/cd838d15487d45ae99751bc557b4a742 => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:04:58.501729 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:58.509600 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:58.631924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:04:58.649895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3cc654-0a95-498b-9f63-09b44a9c0e87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:58.650510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3cc654-0a95-498b-9f63-09b44a9c0e87 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:58.650798 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3cc654-0a95-498b-9f63-09b44a9c0e87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:58.651074 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3cc654-0a95-498b-9f63-09b44a9c0e87 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:58.651724 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3cc654-0a95-498b-9f63-09b44a9c0e87 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:58.653894 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3cc654-0a95-498b-9f63-09b44a9c0e87 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:58.672120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae2ab01-3efd-4aa8-a541-3fe1aae4e97e admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:58.678536 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bfae91-b981-424f-a107-6e1b526b1264 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:58.680278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bfae91-b981-424f-a107-6e1b526b1264 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:58.680786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bfae91-b981-424f-a107-6e1b526b1264 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:58.681730 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bfae91-b981-424f-a107-6e1b526b1264 admin admin] PATH_INFO: `/v3/users/4204338c7de24eaca2d191e9da3ec02d` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:58.684068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bfae91-b981-424f-a107-6e1b526b1264 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4204338c7de24eaca2d191e9da3ec02d)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:58.686597 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59bfae91-b981-424f-a107-6e1b526b1264 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:58.846064 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 817/1645] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:04:58 2022] POST /identity/v3/projects => generated 403 bytes in 435 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:58.847576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3cc654-0a95-498b-9f63-09b44a9c0e87 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:58.849903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2a3cc654-0a95-498b-9f63-09b44a9c0e87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:58.986849 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-59bfae91-b981-424f-a107-6e1b526b1264 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4204338c7de24eaca2d191e9da3ec02d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:59.049562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3cc654-0a95-498b-9f63-09b44a9c0e87 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:59.058620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312d5a18-6398-4f11-8225-957fca981e1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:59.059406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312d5a18-6398-4f11-8225-957fca981e1f admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:59.059798 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312d5a18-6398-4f11-8225-957fca981e1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:59.061000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312d5a18-6398-4f11-8225-957fca981e1f admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:59.061000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312d5a18-6398-4f11-8225-957fca981e1f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:59.063908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312d5a18-6398-4f11-8225-957fca981e1f admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:59.102418 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-59bfae91-b981-424f-a107-6e1b526b1264 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4204338c7de24eaca2d191e9da3ec02d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:59.125820 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-59bfae91-b981-424f-a107-6e1b526b1264 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4204338c7de24eaca2d191e9da3ec02d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:59.128345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-59bfae91-b981-424f-a107-6e1b526b1264 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4204338c7de24eaca2d191e9da3ec02d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:59.147606 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-59bfae91-b981-424f-a107-6e1b526b1264 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4204338c7de24eaca2d191e9da3ec02d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:59.158235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 818/1646] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:04:58 2022] POST /identity/v3/users => generated 308 bytes in 311 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:04:59.159837 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312d5a18-6398-4f11-8225-957fca981e1f admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:59.163795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-312d5a18-6398-4f11-8225-957fca981e1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:59.194193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 829/1647] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:04:58 2022] DELETE /identity/v3/users/4204338c7de24eaca2d191e9da3ec02d => generated 0 bytes in 693 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:04:59.200301 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bfae91-b981-424f-a107-6e1b526b1264 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:59.204526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-59bfae91-b981-424f-a107-6e1b526b1264 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:59.222225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312d5a18-6398-4f11-8225-957fca981e1f admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:59.391506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c12ec4-a1a1-49a2-b96d-bc5dacc70bf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:59.392960 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59bfae91-b981-424f-a107-6e1b526b1264 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:59.393892 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c12ec4-a1a1-49a2-b96d-bc5dacc70bf3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:59.394381 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c12ec4-a1a1-49a2-b96d-bc5dacc70bf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:59.399497 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c12ec4-a1a1-49a2-b96d-bc5dacc70bf3 admin admin] PATH_INFO: `/v3/projects/959a493c8f8e49c990227d2229946a4f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:59.402464 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c12ec4-a1a1-49a2-b96d-bc5dacc70bf3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=959a493c8f8e49c990227d2229946a4f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:59.408121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c12ec4-a1a1-49a2-b96d-bc5dacc70bf3 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:59.415398 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d8345a-d313-449b-a070-d976114577e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:59.416070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d8345a-d313-449b-a070-d976114577e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:59.416354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d8345a-d313-449b-a070-d976114577e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:59.416631 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d8345a-d313-449b-a070-d976114577e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:59.417355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d8345a-d313-449b-a070-d976114577e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:59.418991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d8345a-d313-449b-a070-d976114577e3 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:59.441800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 830/1648] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:59 2022] GET /identity/v3/roles => generated 1420 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:59.443435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d8345a-d313-449b-a070-d976114577e3 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:59.467955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d8345a-d313-449b-a070-d976114577e3 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:59.477650 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef979d27-bfb4-4fd1-8557-89c23b46d1d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:59.478602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef979d27-bfb4-4fd1-8557-89c23b46d1d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:59.479031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef979d27-bfb4-4fd1-8557-89c23b46d1d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:59.479476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef979d27-bfb4-4fd1-8557-89c23b46d1d9 admin admin] PATH_INFO: `/v3/projects/d19b79b17abb4d78a4cc302a316f3688/users/95a5add16c664d53b6c65f9469212093/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:59.516172 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef979d27-bfb4-4fd1-8557-89c23b46d1d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d19b79b17abb4d78a4cc302a316f3688, user_id=95a5add16c664d53b6c65f9469212093, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:59.519293 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef979d27-bfb4-4fd1-8557-89c23b46d1d9 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:59.680279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 831/1649] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:04:59 2022] PUT /identity/v3/projects/d19b79b17abb4d78a4cc302a316f3688/users/95a5add16c664d53b6c65f9469212093/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:04:59.685188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef979d27-bfb4-4fd1-8557-89c23b46d1d9 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:59.729542 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-29c12ec4-a1a1-49a2-b96d-bc5dacc70bf3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '959a493c8f8e49c990227d2229946a4f', 'request_id': 'req-29c12ec4-a1a1-49a2-b96d-bc5dacc70bf3'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:59.748114 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-29c12ec4-a1a1-49a2-b96d-bc5dacc70bf3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '959a493c8f8e49c990227d2229946a4f', 'request_id': 'req-29c12ec4-a1a1-49a2-b96d-bc5dacc70bf3'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:59.817424 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef979d27-bfb4-4fd1-8557-89c23b46d1d9 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:59.844475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-29c12ec4-a1a1-49a2-b96d-bc5dacc70bf3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '959a493c8f8e49c990227d2229946a4f', 'request_id': 'req-29c12ec4-a1a1-49a2-b96d-bc5dacc70bf3'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:59.866960 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-29c12ec4-a1a1-49a2-b96d-bc5dacc70bf3 admin admin] The token cache is being invalidate because project 959a493c8f8e49c990227d2229946a4f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:04:59.867656 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-29c12ec4-a1a1-49a2-b96d-bc5dacc70bf3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:59.869261 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-29c12ec4-a1a1-49a2-b96d-bc5dacc70bf3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:04:59.874305 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 819/1650] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:04:59 2022] DELETE /identity/v3/projects/959a493c8f8e49c990227d2229946a4f => generated 0 bytes in 716 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:04:59.879062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dab932a-1688-462d-b917-25dcecd772a6 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:59.879761 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dab932a-1688-462d-b917-25dcecd772a6 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:59.880985 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dab932a-1688-462d-b917-25dcecd772a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:59.889414 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a2975b-a6ff-4d25-8332-d7eeb25c1d93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:59.890060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a2975b-a6ff-4d25-8332-d7eeb25c1d93 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:59.890339 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a2975b-a6ff-4d25-8332-d7eeb25c1d93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:59.890613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a2975b-a6ff-4d25-8332-d7eeb25c1d93 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:59.891279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a2975b-a6ff-4d25-8332-d7eeb25c1d93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:59.892921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a2975b-a6ff-4d25-8332-d7eeb25c1d93 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:59.922620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 832/1651] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:04:59 2022] GET /identity/v3/roles => generated 1420 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:04:59.929008 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-2dab932a-1688-462d-b917-25dcecd772a6 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:59.929599 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a2975b-a6ff-4d25-8332-d7eeb25c1d93 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:04:59.930039 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-96a2975b-a6ff-4d25-8332-d7eeb25c1d93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:59.930288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2dab932a-1688-462d-b917-25dcecd772a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:04:59.949516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 820/1652] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:04:59.953431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a2975b-a6ff-4d25-8332-d7eeb25c1d93 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:04:59.955722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5935cd-c1ee-4f4e-899f-43d1b325adb1 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:59.956088 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5935cd-c1ee-4f4e-899f-43d1b325adb1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:59.956371 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5935cd-c1ee-4f4e-899f-43d1b325adb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:59.969124 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e2c5ac-1592-45f6-a2c0-110796c7aa61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:04:59.969816 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e2c5ac-1592-45f6-a2c0-110796c7aa61 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:04:59.970101 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e2c5ac-1592-45f6-a2c0-110796c7aa61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:04:59.970380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e2c5ac-1592-45f6-a2c0-110796c7aa61 admin admin] PATH_INFO: `/v3/projects/d19b79b17abb4d78a4cc302a316f3688/users/95a5add16c664d53b6c65f9469212093/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:04:59.973030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e2c5ac-1592-45f6-a2c0-110796c7aa61 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d19b79b17abb4d78a4cc302a316f3688, user_id=95a5add16c664d53b6c65f9469212093, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:04:59.981517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e2c5ac-1592-45f6-a2c0-110796c7aa61 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:04:59.993375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-4f5935cd-c1ee-4f4e-899f-43d1b325adb1 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:04:59.994612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-4f5935cd-c1ee-4f4e-899f-43d1b325adb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:00.006345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 833/1653] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:04:59 2022] PUT /identity/v3/projects/d19b79b17abb4d78a4cc302a316f3688/users/95a5add16c664d53b6c65f9469212093/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:05:00.006692 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e2c5ac-1592-45f6-a2c0-110796c7aa61 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:00.015763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-18e2c5ac-1592-45f6-a2c0-110796c7aa61 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:00.141140 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 821/1654] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:04:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:05:00.144492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5935cd-c1ee-4f4e-899f-43d1b325adb1 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:00.160443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:00.172103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e2c5ac-1592-45f6-a2c0-110796c7aa61 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:00.172307 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5935cd-c1ee-4f4e-899f-43d1b325adb1 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:00.177544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5821a2-0fba-4e8e-bae7-880a879558f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:00.178752 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5821a2-0fba-4e8e-bae7-880a879558f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:00.179088 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5821a2-0fba-4e8e-bae7-880a879558f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:00.180144 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5821a2-0fba-4e8e-bae7-880a879558f6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:00.180833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5821a2-0fba-4e8e-bae7-880a879558f6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:05:00.181177 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5821a2-0fba-4e8e-bae7-880a879558f6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:00.182817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5821a2-0fba-4e8e-bae7-880a879558f6 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:00.184892 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442ef28f-821e-45a3-a9f5-8001702ee302 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:00.185606 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442ef28f-821e-45a3-a9f5-8001702ee302 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:00.185923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442ef28f-821e-45a3-a9f5-8001702ee302 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:00.186132 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442ef28f-821e-45a3-a9f5-8001702ee302 admin admin] PATH_INFO: `/v3/projects/ee5bd28fea9e4ab49b23f0ececbe3e26` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:00.187794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442ef28f-821e-45a3-a9f5-8001702ee302 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee5bd28fea9e4ab49b23f0ececbe3e26)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:00.190139 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442ef28f-821e-45a3-a9f5-8001702ee302 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:00.214905 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 822/1655] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:05:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:05:00.216965 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5821a2-0fba-4e8e-bae7-880a879558f6 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:00.241391 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5821a2-0fba-4e8e-bae7-880a879558f6 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:00.246934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:00.252220 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bc64fd-f18d-41c4-bc46-533574d9a141 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:00.252899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bc64fd-f18d-41c4-bc46-533574d9a141 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:00.253848 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bc64fd-f18d-41c4-bc46-533574d9a141 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:00.253848 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bc64fd-f18d-41c4-bc46-533574d9a141 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:00.254253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bc64fd-f18d-41c4-bc46-533574d9a141 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:00.256847 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bc64fd-f18d-41c4-bc46-533574d9a141 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:00.369739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 823/1656] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:00 2022] POST /identity/v3/projects => generated 371 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:00.374632 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bc64fd-f18d-41c4-bc46-533574d9a141 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:00.391502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-442ef28f-821e-45a3-a9f5-8001702ee302 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee5bd28fea9e4ab49b23f0ececbe3e26', 'request_id': 'req-442ef28f-821e-45a3-a9f5-8001702ee302'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:00.407183 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-442ef28f-821e-45a3-a9f5-8001702ee302 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee5bd28fea9e4ab49b23f0ececbe3e26', 'request_id': 'req-442ef28f-821e-45a3-a9f5-8001702ee302'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:00.420299 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-442ef28f-821e-45a3-a9f5-8001702ee302 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee5bd28fea9e4ab49b23f0ececbe3e26', 'request_id': 'req-442ef28f-821e-45a3-a9f5-8001702ee302'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:00.515773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-442ef28f-821e-45a3-a9f5-8001702ee302 admin admin] The token cache is being invalidate because project ee5bd28fea9e4ab49b23f0ececbe3e26 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:05:00.515878 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-442ef28f-821e-45a3-a9f5-8001702ee302 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:00.516636 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-442ef28f-821e-45a3-a9f5-8001702ee302 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:00.518749 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 834/1657] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:05:00 2022] DELETE /identity/v3/projects/ee5bd28fea9e4ab49b23f0ececbe3e26 => generated 0 bytes in 513 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:00.525271 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b33b874-b0a8-4959-8c85-6e97e3043218 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:00.525570 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b33b874-b0a8-4959-8c85-6e97e3043218 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:00.526529 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b33b874-b0a8-4959-8c85-6e97e3043218 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:00.535037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bc64fd-f18d-41c4-bc46-533574d9a141 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:00.550358 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd208cba-b4f1-41aa-aa48-7b7e11fef206 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:00.551265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd208cba-b4f1-41aa-aa48-7b7e11fef206 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:00.551717 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd208cba-b4f1-41aa-aa48-7b7e11fef206 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:00.553154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd208cba-b4f1-41aa-aa48-7b7e11fef206 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:00.559010 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd208cba-b4f1-41aa-aa48-7b7e11fef206 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:00.561649 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd208cba-b4f1-41aa-aa48-7b7e11fef206 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:00.590670 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-5b33b874-b0a8-4959-8c85-6e97e3043218 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:00.592234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-5b33b874-b0a8-4959-8c85-6e97e3043218 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:00.625153 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 835/1658] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:00.629831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4505ed-48d4-4ac5-bd9b-74f400821c27 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:00.630197 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4505ed-48d4-4ac5-bd9b-74f400821c27 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:00.630471 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4505ed-48d4-4ac5-bd9b-74f400821c27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:00.671032 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 824/1659] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:00 2022] POST /identity/v3/users => generated 286 bytes in 297 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:00.674549 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-3b4505ed-48d4-4ac5-bd9b-74f400821c27 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:00.676284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd208cba-b4f1-41aa-aa48-7b7e11fef206 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:00.678111 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-3b4505ed-48d4-4ac5-bd9b-74f400821c27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:00.680003 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-fd208cba-b4f1-41aa-aa48-7b7e11fef206 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:00.732167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 836/1660] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:05:00.733492 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4505ed-48d4-4ac5-bd9b-74f400821c27 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:00.740712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd208cba-b4f1-41aa-aa48-7b7e11fef206 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:00.752528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ceefb5-ce0d-4708-94dc-9ef1f0741458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:00.753674 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ceefb5-ce0d-4708-94dc-9ef1f0741458 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:00.754202 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ceefb5-ce0d-4708-94dc-9ef1f0741458 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:00.754687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ceefb5-ce0d-4708-94dc-9ef1f0741458 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:00.756178 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ceefb5-ce0d-4708-94dc-9ef1f0741458 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:00.759851 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ceefb5-ce0d-4708-94dc-9ef1f0741458 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:00.774116 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4505ed-48d4-4ac5-bd9b-74f400821c27 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:00.780950 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 825/1661] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:00 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:00.787311 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ceefb5-ce0d-4708-94dc-9ef1f0741458 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:00.791147 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434d9fb3-57f7-405b-a0d0-c427b7858314 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:00.791629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434d9fb3-57f7-405b-a0d0-c427b7858314 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:00.791958 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434d9fb3-57f7-405b-a0d0-c427b7858314 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:00.792245 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434d9fb3-57f7-405b-a0d0-c427b7858314 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:00.795600 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434d9fb3-57f7-405b-a0d0-c427b7858314 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:05:00.795944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434d9fb3-57f7-405b-a0d0-c427b7858314 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:00.799796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434d9fb3-57f7-405b-a0d0-c427b7858314 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:00.810274 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ceefb5-ce0d-4708-94dc-9ef1f0741458 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:00.819900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 837/1662] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:05:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:05:00.824112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434d9fb3-57f7-405b-a0d0-c427b7858314 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:00.834229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a923fc7f-a4c8-46c5-b697-2555bac52f8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:00.835289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a923fc7f-a4c8-46c5-b697-2555bac52f8b admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:00.835763 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a923fc7f-a4c8-46c5-b697-2555bac52f8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:00.836126 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a923fc7f-a4c8-46c5-b697-2555bac52f8b admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:00.838008 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a923fc7f-a4c8-46c5-b697-2555bac52f8b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:00.841806 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a923fc7f-a4c8-46c5-b697-2555bac52f8b admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:00.846457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434d9fb3-57f7-405b-a0d0-c427b7858314 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:00.864871 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bf2bee-cdf3-4df7-9808-fea097d911c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:00.865617 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bf2bee-cdf3-4df7-9808-fea097d911c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:00.866026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bf2bee-cdf3-4df7-9808-fea097d911c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:00.866364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bf2bee-cdf3-4df7-9808-fea097d911c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:00.867485 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48bf2bee-cdf3-4df7-9808-fea097d911c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:00.871959 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48bf2bee-cdf3-4df7-9808-fea097d911c1 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:00.878142 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 826/1663] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:00 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:00.880671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a923fc7f-a4c8-46c5-b697-2555bac52f8b admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:00.904887 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a923fc7f-a4c8-46c5-b697-2555bac52f8b admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:00.930243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8d50cf-2ba4-4268-857e-bb611739d8bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:00.930243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8d50cf-2ba4-4268-857e-bb611739d8bd admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:00.930243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8d50cf-2ba4-4268-857e-bb611739d8bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:00.930243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8d50cf-2ba4-4268-857e-bb611739d8bd admin admin] PATH_INFO: `/v3/projects/bac331f898134ea787f4eb29b701b3ea/users/f8b21d05979445989ca7e1a22ec71740/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:00.961342 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8d50cf-2ba4-4268-857e-bb611739d8bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=bac331f898134ea787f4eb29b701b3ea, user_id=f8b21d05979445989ca7e1a22ec71740, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:00.964682 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8d50cf-2ba4-4268-857e-bb611739d8bd admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:01.097939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 838/1664] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:00 2022] POST /identity/v3/projects => generated 391 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:05:01.102865 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bf2bee-cdf3-4df7-9808-fea097d911c1 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:01.147806 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 827/1665] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:00 2022] PUT /identity/v3/projects/bac331f898134ea787f4eb29b701b3ea/users/f8b21d05979445989ca7e1a22ec71740/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:01.149699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b48b80-f6f4-43f9-886c-10d4bca7b912 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:01.150198 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b48b80-f6f4-43f9-886c-10d4bca7b912 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:01.150651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b48b80-f6f4-43f9-886c-10d4bca7b912 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:01.215381 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-97b48b80-f6f4-43f9-886c-10d4bca7b912 None None] MFA Rules not processed for user `f8b21d05979445989ca7e1a22ec71740`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:01.222331 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-97b48b80-f6f4-43f9-886c-10d4bca7b912 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:01.254389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bf2bee-cdf3-4df7-9808-fea097d911c1 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:01.360223 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 828/1666] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:01 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:01.368523 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b48b80-f6f4-43f9-886c-10d4bca7b912 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:01.388515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beafc4ee-686e-42be-97c5-ddf368eddd08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:01.392262 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beafc4ee-686e-42be-97c5-ddf368eddd08 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:01.392262 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beafc4ee-686e-42be-97c5-ddf368eddd08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:01.392262 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beafc4ee-686e-42be-97c5-ddf368eddd08 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:01.392262 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beafc4ee-686e-42be-97c5-ddf368eddd08 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:01.392514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beafc4ee-686e-42be-97c5-ddf368eddd08 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:01.403702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b48b80-f6f4-43f9-886c-10d4bca7b912 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:01.415788 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8782ab1f-581e-4e78-8956-7f208441a238 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:01.416515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8782ab1f-581e-4e78-8956-7f208441a238 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:01.419414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8782ab1f-581e-4e78-8956-7f208441a238 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:01.419414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8782ab1f-581e-4e78-8956-7f208441a238 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:01.419414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8782ab1f-581e-4e78-8956-7f208441a238 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:01.421246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8782ab1f-581e-4e78-8956-7f208441a238 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:01.510076 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 839/1667] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:01 2022] POST /identity/v3/users => generated 296 bytes in 408 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:01.514187 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beafc4ee-686e-42be-97c5-ddf368eddd08 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:01.545291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 829/1668] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:01 2022] POST /identity/v3/projects => generated 371 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:01.545470 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beafc4ee-686e-42be-97c5-ddf368eddd08 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:01.546519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8782ab1f-581e-4e78-8956-7f208441a238 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:01.673739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f670dc4-446d-4b98-8425-55bb9df5602b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:01.674364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f670dc4-446d-4b98-8425-55bb9df5602b admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:01.674640 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f670dc4-446d-4b98-8425-55bb9df5602b admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:01.674913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f670dc4-446d-4b98-8425-55bb9df5602b admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:01.675576 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f670dc4-446d-4b98-8425-55bb9df5602b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:01.678115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f670dc4-446d-4b98-8425-55bb9df5602b admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:01.710640 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 840/1669] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:01 2022] GET /identity/v3/roles => generated 1420 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:05:01.710640 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f670dc4-446d-4b98-8425-55bb9df5602b admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:01.710765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8782ab1f-581e-4e78-8956-7f208441a238 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:01.721914 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8aa56a-2501-45b9-b47a-f6d196c4d851 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:01.724311 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8aa56a-2501-45b9-b47a-f6d196c4d851 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:01.726669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8aa56a-2501-45b9-b47a-f6d196c4d851 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:01.726982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8aa56a-2501-45b9-b47a-f6d196c4d851 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:01.727678 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8aa56a-2501-45b9-b47a-f6d196c4d851 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:01.736293 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8aa56a-2501-45b9-b47a-f6d196c4d851 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:01.745915 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f670dc4-446d-4b98-8425-55bb9df5602b admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:01.756436 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846a0a30-b277-4b56-af4d-c1ac49231cfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:01.757061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846a0a30-b277-4b56-af4d-c1ac49231cfc admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:01.757408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846a0a30-b277-4b56-af4d-c1ac49231cfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:01.757841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846a0a30-b277-4b56-af4d-c1ac49231cfc admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:01.758678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846a0a30-b277-4b56-af4d-c1ac49231cfc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:01.760486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846a0a30-b277-4b56-af4d-c1ac49231cfc admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:01.783991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 841/1670] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:01 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:01.788798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846a0a30-b277-4b56-af4d-c1ac49231cfc admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:01.816077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846a0a30-b277-4b56-af4d-c1ac49231cfc admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:01.842845 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe24b1e-5f4d-49e2-89b2-f4987ef59640 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:01.843923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe24b1e-5f4d-49e2-89b2-f4987ef59640 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:01.844499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe24b1e-5f4d-49e2-89b2-f4987ef59640 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:01.844911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe24b1e-5f4d-49e2-89b2-f4987ef59640 admin admin] PATH_INFO: `/v3/projects/2b4087f305b5468b973de4fe953bee24/users/16e735fec04b41839258a8fd87ef7b2b/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:01.880625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe24b1e-5f4d-49e2-89b2-f4987ef59640 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2b4087f305b5468b973de4fe953bee24, user_id=16e735fec04b41839258a8fd87ef7b2b, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:01.883697 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 830/1671] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:01 2022] POST /identity/v3/users => generated 292 bytes in 337 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:01.885067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe24b1e-5f4d-49e2-89b2-f4987ef59640 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:01.887479 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8aa56a-2501-45b9-b47a-f6d196c4d851 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:01.923546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8aa56a-2501-45b9-b47a-f6d196c4d851 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:01.949763 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8392084b-d715-455d-86fb-6f7404f086cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:01.950437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8392084b-d715-455d-86fb-6f7404f086cd admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:01.950700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8392084b-d715-455d-86fb-6f7404f086cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:01.950968 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8392084b-d715-455d-86fb-6f7404f086cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:01.953197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8392084b-d715-455d-86fb-6f7404f086cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:01.965099 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8392084b-d715-455d-86fb-6f7404f086cd admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:01.979499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 842/1672] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:01 2022] PUT /identity/v3/projects/2b4087f305b5468b973de4fe953bee24/users/16e735fec04b41839258a8fd87ef7b2b/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:01.992356 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd96e71-6721-447f-8d16-66417b240219 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:01.992639 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd96e71-6721-447f-8d16-66417b240219 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:01.992917 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd96e71-6721-447f-8d16-66417b240219 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:02.014630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 831/1673] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:01 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:02.016664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8392084b-d715-455d-86fb-6f7404f086cd admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:02.058513 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8392084b-d715-455d-86fb-6f7404f086cd admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:02.079676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9bd96e71-6721-447f-8d16-66417b240219 None None] MFA Rules not processed for user `16e735fec04b41839258a8fd87ef7b2b`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:02.084110 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9bd96e71-6721-447f-8d16-66417b240219 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:02.141324 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:02.189397 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:02.250305 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566436cd-6af3-48f8-b160-29fd52a7730e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:02.250918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566436cd-6af3-48f8-b160-29fd52a7730e admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:02.251196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566436cd-6af3-48f8-b160-29fd52a7730e admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:02.251473 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566436cd-6af3-48f8-b160-29fd52a7730e admin admin] PATH_INFO: `/v3/projects/f14a3f986b774cb09bd5cc3f9ebeaac3/users/e21c2486e9a342f588402f15d692bd49/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:02.273826 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 843/1674] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:01 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:02.278411 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd96e71-6721-447f-8d16-66417b240219 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:02.282553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566436cd-6af3-48f8-b160-29fd52a7730e admin admin] RBAC: Authorizing `identity:create_grant(project_id=f14a3f986b774cb09bd5cc3f9ebeaac3, user_id=e21c2486e9a342f588402f15d692bd49, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:02.286840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566436cd-6af3-48f8-b160-29fd52a7730e admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:02.314900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd96e71-6721-447f-8d16-66417b240219 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:02.329573 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c2a985-f0ab-4a07-9580-7de5b887ef47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:02.330367 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c2a985-f0ab-4a07-9580-7de5b887ef47 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:02.341787 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c2a985-f0ab-4a07-9580-7de5b887ef47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:02.342429 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c2a985-f0ab-4a07-9580-7de5b887ef47 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:02.343880 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c2a985-f0ab-4a07-9580-7de5b887ef47 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:02.347433 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c2a985-f0ab-4a07-9580-7de5b887ef47 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:02.377373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 832/1675] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:02 2022] PUT /identity/v3/projects/f14a3f986b774cb09bd5cc3f9ebeaac3/users/e21c2486e9a342f588402f15d692bd49/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:02.396479 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566436cd-6af3-48f8-b160-29fd52a7730e admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:02.455264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 844/1676] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:02 2022] POST /identity/v3/projects => generated 393 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:02.463489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c2a985-f0ab-4a07-9580-7de5b887ef47 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:02.637514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566436cd-6af3-48f8-b160-29fd52a7730e admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:02.638063 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c2a985-f0ab-4a07-9580-7de5b887ef47 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:02.646752 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031a1074-6bc9-4e89-a47d-5531f7e0e94e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:02.647606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031a1074-6bc9-4e89-a47d-5531f7e0e94e admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:02.648421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031a1074-6bc9-4e89-a47d-5531f7e0e94e admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:02.648775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031a1074-6bc9-4e89-a47d-5531f7e0e94e admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:02.649977 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031a1074-6bc9-4e89-a47d-5531f7e0e94e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:02.652691 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad4be7d-46a8-4fd2-abb7-07085db0594c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:02.653531 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad4be7d-46a8-4fd2-abb7-07085db0594c admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:02.653985 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad4be7d-46a8-4fd2-abb7-07085db0594c admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:02.654393 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad4be7d-46a8-4fd2-abb7-07085db0594c admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:02.654992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031a1074-6bc9-4e89-a47d-5531f7e0e94e admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:02.655488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad4be7d-46a8-4fd2-abb7-07085db0594c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:02.659282 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad4be7d-46a8-4fd2-abb7-07085db0594c admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:02.675661 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 833/1677] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:02 2022] GET /identity/v3/roles => generated 1420 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:02.680683 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031a1074-6bc9-4e89-a47d-5531f7e0e94e admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:02.717773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031a1074-6bc9-4e89-a47d-5531f7e0e94e admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:02.727364 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159e0278-64aa-4236-8cc5-0e1d0a55b843 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:02.727364 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159e0278-64aa-4236-8cc5-0e1d0a55b843 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:02.735309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159e0278-64aa-4236-8cc5-0e1d0a55b843 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:02.735309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159e0278-64aa-4236-8cc5-0e1d0a55b843 admin admin] PATH_INFO: `/v3/projects/f14a3f986b774cb09bd5cc3f9ebeaac3/users/e21c2486e9a342f588402f15d692bd49/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:02.735309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159e0278-64aa-4236-8cc5-0e1d0a55b843 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f14a3f986b774cb09bd5cc3f9ebeaac3, user_id=e21c2486e9a342f588402f15d692bd49, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:02.737435 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159e0278-64aa-4236-8cc5-0e1d0a55b843 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:02.779002 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 845/1678] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:02 2022] POST /identity/v3/users => generated 303 bytes in 323 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:02.783009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad4be7d-46a8-4fd2-abb7-07085db0594c admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:02.783476 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 834/1679] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:02 2022] PUT /identity/v3/projects/f14a3f986b774cb09bd5cc3f9ebeaac3/users/e21c2486e9a342f588402f15d692bd49/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:02.797035 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e93c0c1-cd27-4a9d-8f78-cc082353cc2c None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:02.800299 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e93c0c1-cd27-4a9d-8f78-cc082353cc2c None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:02.800299 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e93c0c1-cd27-4a9d-8f78-cc082353cc2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:02.812643 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad4be7d-46a8-4fd2-abb7-07085db0594c admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:02.877024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-4e93c0c1-cd27-4a9d-8f78-cc082353cc2c None None] MFA Rules not processed for user `e21c2486e9a342f588402f15d692bd49`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:02.886383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-4e93c0c1-cd27-4a9d-8f78-cc082353cc2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:03.015145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41983a95-2493-4bb9-be02-1ed94b3d05c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:03.015971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41983a95-2493-4bb9-be02-1ed94b3d05c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:03.016473 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41983a95-2493-4bb9-be02-1ed94b3d05c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:03.016843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41983a95-2493-4bb9-be02-1ed94b3d05c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:03.018336 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41983a95-2493-4bb9-be02-1ed94b3d05c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:03.020904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41983a95-2493-4bb9-be02-1ed94b3d05c5 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:03.050318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 846/1680] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:02 2022] GET /identity/v3/roles => generated 1420 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:03.055220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41983a95-2493-4bb9-be02-1ed94b3d05c5 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:03.076487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 835/1681] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:02 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:03.077385 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e93c0c1-cd27-4a9d-8f78-cc082353cc2c None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:03.081283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-4e93c0c1-cd27-4a9d-8f78-cc082353cc2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:03.090603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41983a95-2493-4bb9-be02-1ed94b3d05c5 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:03.113945 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a286ab1e-8914-4632-9c2d-01caed0009d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:03.115950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a286ab1e-8914-4632-9c2d-01caed0009d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:03.118430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a286ab1e-8914-4632-9c2d-01caed0009d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:03.118837 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a286ab1e-8914-4632-9c2d-01caed0009d5 admin admin] PATH_INFO: `/v3/projects/42de058800864cb7b2e062ad4bd1fd46/users/2fb5f8174ff345519dbbf97c156972dc/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:03.136089 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:03.166243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:03.208277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a286ab1e-8914-4632-9c2d-01caed0009d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=42de058800864cb7b2e062ad4bd1fd46, user_id=2fb5f8174ff345519dbbf97c156972dc, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:03.220677 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a286ab1e-8914-4632-9c2d-01caed0009d5 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:03.257364 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e93c0c1-cd27-4a9d-8f78-cc082353cc2c None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:03.261737 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da63be9-aa0e-4cf6-ab4f-25f874adbc91 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:03.262622 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da63be9-aa0e-4cf6-ab4f-25f874adbc91 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:03.263047 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da63be9-aa0e-4cf6-ab4f-25f874adbc91 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:03.263461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da63be9-aa0e-4cf6-ab4f-25f874adbc91 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:03.287514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da63be9-aa0e-4cf6-ab4f-25f874adbc91 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:03.289319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da63be9-aa0e-4cf6-ab4f-25f874adbc91 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:03.314854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 836/1682] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:03 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:03.316709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd4644d-f147-422e-a153-5144c23e95da None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:03.317254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd4644d-f147-422e-a153-5144c23e95da None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:03.317730 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd4644d-f147-422e-a153-5144c23e95da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:03.346965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 847/1683] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:03 2022] PUT /identity/v3/projects/42de058800864cb7b2e062ad4bd1fd46/users/2fb5f8174ff345519dbbf97c156972dc/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:05:03.351874 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a286ab1e-8914-4632-9c2d-01caed0009d5 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:03.480554 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-fbd4644d-f147-422e-a153-5144c23e95da None None] MFA Rules not processed for user `95a5add16c664d53b6c65f9469212093`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:03.482373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-fbd4644d-f147-422e-a153-5144c23e95da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:03.598437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a286ab1e-8914-4632-9c2d-01caed0009d5 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:03.614016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef781798-a874-40c7-ba77-03b6320fb3eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:03.614916 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef781798-a874-40c7-ba77-03b6320fb3eb admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:03.615362 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef781798-a874-40c7-ba77-03b6320fb3eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:03.615773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef781798-a874-40c7-ba77-03b6320fb3eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:03.616570 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef781798-a874-40c7-ba77-03b6320fb3eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:03.618479 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef781798-a874-40c7-ba77-03b6320fb3eb admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:03.651121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 837/1684] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:03 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 335 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:03.658324 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 848/1685] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:03 2022] GET /identity/v3/roles => generated 1420 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:05:03.660111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd4644d-f147-422e-a153-5144c23e95da None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:03.680342 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd4644d-f147-422e-a153-5144c23e95da None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:03.686088 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe88259-a06e-4c18-953c-eb9d00da55d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:03.687187 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe88259-a06e-4c18-953c-eb9d00da55d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:03.687572 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe88259-a06e-4c18-953c-eb9d00da55d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:03.687948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe88259-a06e-4c18-953c-eb9d00da55d6 admin admin] PATH_INFO: `/v3/projects/42de058800864cb7b2e062ad4bd1fd46/users/2fb5f8174ff345519dbbf97c156972dc/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:03.691147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe88259-a06e-4c18-953c-eb9d00da55d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=42de058800864cb7b2e062ad4bd1fd46, user_id=2fb5f8174ff345519dbbf97c156972dc, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:03.696640 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe88259-a06e-4c18-953c-eb9d00da55d6 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:03.738353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 838/1686] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:03 2022] PUT /identity/v3/projects/42de058800864cb7b2e062ad4bd1fd46/users/2fb5f8174ff345519dbbf97c156972dc/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:03.749003 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5499f2f5-60b5-4826-8e3b-585985fa82d2 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:03.749562 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5499f2f5-60b5-4826-8e3b-585985fa82d2 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:03.750037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5499f2f5-60b5-4826-8e3b-585985fa82d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:03.806082 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-5499f2f5-60b5-4826-8e3b-585985fa82d2 None None] MFA Rules not processed for user `2fb5f8174ff345519dbbf97c156972dc`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:03.807624 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-5499f2f5-60b5-4826-8e3b-585985fa82d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:03.856664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:04.061781 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 849/1687] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:03 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 314 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:04.084098 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe88259-a06e-4c18-953c-eb9d00da55d6 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:04.216575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:04.266321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe88259-a06e-4c18-953c-eb9d00da55d6 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:04.271819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792d513c-69b0-4f25-9aee-fdf63e2944b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:04.272505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792d513c-69b0-4f25-9aee-fdf63e2944b0 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:04.272785 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792d513c-69b0-4f25-9aee-fdf63e2944b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:04.273154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792d513c-69b0-4f25-9aee-fdf63e2944b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:04.299408 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792d513c-69b0-4f25-9aee-fdf63e2944b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:04.300929 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792d513c-69b0-4f25-9aee-fdf63e2944b0 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:04.320875 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 839/1688] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:04 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:04.493927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5499f2f5-60b5-4826-8e3b-585985fa82d2 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:04.508975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5499f2f5-60b5-4826-8e3b-585985fa82d2 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:04.522957 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f4b101-a825-4e6d-9c77-8b6798270073 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:04.523592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f4b101-a825-4e6d-9c77-8b6798270073 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:04.523871 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f4b101-a825-4e6d-9c77-8b6798270073 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:04.524141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f4b101-a825-4e6d-9c77-8b6798270073 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:04.664188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f4b101-a825-4e6d-9c77-8b6798270073 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:04.667310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f4b101-a825-4e6d-9c77-8b6798270073 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:04.685636 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 850/1689] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:04 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:05.016018 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792d513c-69b0-4f25-9aee-fdf63e2944b0 service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:05.174045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:05.181413 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792d513c-69b0-4f25-9aee-fdf63e2944b0 service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:05.197919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7754f32a-7fce-4e8b-a8d5-c2ed11f276d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:05.198588 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7754f32a-7fce-4e8b-a8d5-c2ed11f276d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:05.198874 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7754f32a-7fce-4e8b-a8d5-c2ed11f276d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:05.199153 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7754f32a-7fce-4e8b-a8d5-c2ed11f276d3 admin admin] PATH_INFO: `/v3/users/f8b21d05979445989ca7e1a22ec71740` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:05.200362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7754f32a-7fce-4e8b-a8d5-c2ed11f276d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f8b21d05979445989ca7e1a22ec71740)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:05.203784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7754f32a-7fce-4e8b-a8d5-c2ed11f276d3 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:05.260026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:05.477144 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7754f32a-7fce-4e8b-a8d5-c2ed11f276d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f8b21d05979445989ca7e1a22ec71740'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:05.477993 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7754f32a-7fce-4e8b-a8d5-c2ed11f276d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f8b21d05979445989ca7e1a22ec71740'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:05.547485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7754f32a-7fce-4e8b-a8d5-c2ed11f276d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f8b21d05979445989ca7e1a22ec71740'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:05.574150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7754f32a-7fce-4e8b-a8d5-c2ed11f276d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f8b21d05979445989ca7e1a22ec71740'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:05.575764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7754f32a-7fce-4e8b-a8d5-c2ed11f276d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f8b21d05979445989ca7e1a22ec71740'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:05.633914 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 840/1690] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:05:05 2022] DELETE /identity/v3/users/f8b21d05979445989ca7e1a22ec71740 => generated 0 bytes in 615 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:05.635674 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55a6fd2-44f9-46e9-ac03-caa590804262 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:05.636455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55a6fd2-44f9-46e9-ac03-caa590804262 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:05.636743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55a6fd2-44f9-46e9-ac03-caa590804262 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:05.692690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-c55a6fd2-44f9-46e9-ac03-caa590804262 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:05.692690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c55a6fd2-44f9-46e9-ac03-caa590804262 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:05.847929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 851/1691] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:06.044344 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7754f32a-7fce-4e8b-a8d5-c2ed11f276d3 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:06.045942 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-7754f32a-7fce-4e8b-a8d5-c2ed11f276d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:06.182572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d97251-4ffd-464b-b097-4fd55b0bae54 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:06.183075 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d97251-4ffd-464b-b097-4fd55b0bae54 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:06.183514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d97251-4ffd-464b-b097-4fd55b0bae54 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:06.185408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 852/1692] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 4 01:05:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:05:06.198448 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7754f32a-7fce-4e8b-a8d5-c2ed11f276d3 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:06.211864 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67f954b-efb8-452a-a0b5-fada95920028 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:06.212516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67f954b-efb8-452a-a0b5-fada95920028 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:06.212796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67f954b-efb8-452a-a0b5-fada95920028 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:06.213066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67f954b-efb8-452a-a0b5-fada95920028 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:06.238573 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67f954b-efb8-452a-a0b5-fada95920028 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:06.240766 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67f954b-efb8-452a-a0b5-fada95920028 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:06.264081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 841/1693] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:06.266456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7b9dd4-c6fd-4623-a47e-62914ccc8605 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:06.266889 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7b9dd4-c6fd-4623-a47e-62914ccc8605 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:06.267163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7b9dd4-c6fd-4623-a47e-62914ccc8605 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:06.309355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9a7b9dd4-c6fd-4623-a47e-62914ccc8605 None None] MFA Rules not processed for user `b29f444d56ec49308f7fef16f4d1eea5`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:06.311742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9a7b9dd4-c6fd-4623-a47e-62914ccc8605 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:06.418493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:06.464798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 853/1694] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 4 01:05:06 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:05:06.499474 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67f954b-efb8-452a-a0b5-fada95920028 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:06.502715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f67f954b-efb8-452a-a0b5-fada95920028 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:06.545503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67f954b-efb8-452a-a0b5-fada95920028 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:06.553474 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b093700e-456e-419a-8747-75a30b6a9df8 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b29f444d56ec49308f7fef16f4d1eea5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:06.554096 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b093700e-456e-419a-8747-75a30b6a9df8 service placement] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:06.554535 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b093700e-456e-419a-8747-75a30b6a9df8 service placement] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:06.554935 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b093700e-456e-419a-8747-75a30b6a9df8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:06.569662 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b093700e-456e-419a-8747-75a30b6a9df8 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:06.573138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b093700e-456e-419a-8747-75a30b6a9df8 service placement] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:06.588981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 842/1695] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:06 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:07.490466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7b9dd4-c6fd-4623-a47e-62914ccc8605 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:07.491912 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9a7b9dd4-c6fd-4623-a47e-62914ccc8605 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:07.519319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7b9dd4-c6fd-4623-a47e-62914ccc8605 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:07.532440 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f6d574-0fcd-4f6e-8a2f-9e64981e6ca9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:07.533125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f6d574-0fcd-4f6e-8a2f-9e64981e6ca9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:07.533408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f6d574-0fcd-4f6e-8a2f-9e64981e6ca9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:07.533680 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f6d574-0fcd-4f6e-8a2f-9e64981e6ca9 admin admin] PATH_INFO: `/v3/projects/bac331f898134ea787f4eb29b701b3ea` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:07.534852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f6d574-0fcd-4f6e-8a2f-9e64981e6ca9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bac331f898134ea787f4eb29b701b3ea)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:07.542336 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f6d574-0fcd-4f6e-8a2f-9e64981e6ca9 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:07.746042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-62f6d574-0fcd-4f6e-8a2f-9e64981e6ca9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bac331f898134ea787f4eb29b701b3ea', 'request_id': 'req-62f6d574-0fcd-4f6e-8a2f-9e64981e6ca9'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:07.760626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-62f6d574-0fcd-4f6e-8a2f-9e64981e6ca9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bac331f898134ea787f4eb29b701b3ea', 'request_id': 'req-62f6d574-0fcd-4f6e-8a2f-9e64981e6ca9'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:07.774107 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-62f6d574-0fcd-4f6e-8a2f-9e64981e6ca9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bac331f898134ea787f4eb29b701b3ea', 'request_id': 'req-62f6d574-0fcd-4f6e-8a2f-9e64981e6ca9'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:07.852798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-62f6d574-0fcd-4f6e-8a2f-9e64981e6ca9 admin admin] The token cache is being invalidate because project bac331f898134ea787f4eb29b701b3ea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:05:07.853485 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-62f6d574-0fcd-4f6e-8a2f-9e64981e6ca9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:07.854388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-62f6d574-0fcd-4f6e-8a2f-9e64981e6ca9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:07.859710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 854/1696] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:05:07 2022] DELETE /identity/v3/projects/bac331f898134ea787f4eb29b701b3ea => generated 0 bytes in 367 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:07.860727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b093700e-456e-419a-8747-75a30b6a9df8 service placement] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:07.875703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b093700e-456e-419a-8747-75a30b6a9df8 service placement] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:07.904861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c846b2a7-6e6a-4ed3-8b63-937253728a12 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:07.905288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c846b2a7-6e6a-4ed3-8b63-937253728a12 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:07.905626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c846b2a7-6e6a-4ed3-8b63-937253728a12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:07.989922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-c846b2a7-6e6a-4ed3-8b63-937253728a12 None None] MFA Rules not processed for user `b6b95b6814cf4cd8b77d87e7810c70e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:07.991195 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c846b2a7-6e6a-4ed3-8b63-937253728a12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:08.041722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e32466e-5bc7-4c48-941f-5526d429d82f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:08.042439 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e32466e-5bc7-4c48-941f-5526d429d82f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:08.042727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e32466e-5bc7-4c48-941f-5526d429d82f admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:08.043009 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e32466e-5bc7-4c48-941f-5526d429d82f admin admin] PATH_INFO: `/v3/users/e21c2486e9a342f588402f15d692bd49` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:08.044212 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e32466e-5bc7-4c48-941f-5526d429d82f admin admin] RBAC: Authorizing `identity:delete_user(user_id=e21c2486e9a342f588402f15d692bd49)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:08.046345 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e32466e-5bc7-4c48-941f-5526d429d82f admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:08.057158 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:08.220977 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 855/1697] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:07 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:08.234314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45aebee-8216-4260-bde5-bebded084251 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:08.234314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45aebee-8216-4260-bde5-bebded084251 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:08.234314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45aebee-8216-4260-bde5-bebded084251 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:08.260974 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-a45aebee-8216-4260-bde5-bebded084251 None None] MFA Rules not processed for user `b6b95b6814cf4cd8b77d87e7810c70e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:08.263568 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a45aebee-8216-4260-bde5-bebded084251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:08.274605 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 856/1698] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:08 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:05:08.288279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45aebee-8216-4260-bde5-bebded084251 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:08.293293 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a45aebee-8216-4260-bde5-bebded084251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:08.362003 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5e32466e-5bc7-4c48-941f-5526d429d82f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e21c2486e9a342f588402f15d692bd49'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:08.363621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5e32466e-5bc7-4c48-941f-5526d429d82f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e21c2486e9a342f588402f15d692bd49'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:08.386353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:08.444571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5e32466e-5bc7-4c48-941f-5526d429d82f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e21c2486e9a342f588402f15d692bd49'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:08.456094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45aebee-8216-4260-bde5-bebded084251 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:08.469107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5e32466e-5bc7-4c48-941f-5526d429d82f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e21c2486e9a342f588402f15d692bd49'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:08.470813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5e32466e-5bc7-4c48-941f-5526d429d82f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e21c2486e9a342f588402f15d692bd49'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:08.471347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff10d611-cd64-4cbe-bf4d-b4303bc4e7d9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:08.472539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff10d611-cd64-4cbe-bf4d-b4303bc4e7d9 service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:08.472779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff10d611-cd64-4cbe-bf4d-b4303bc4e7d9 service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:08.473364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff10d611-cd64-4cbe-bf4d-b4303bc4e7d9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:08.478434 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ff10d611-cd64-4cbe-bf4d-b4303bc4e7d9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:08.501312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff10d611-cd64-4cbe-bf4d-b4303bc4e7d9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:08.506337 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff10d611-cd64-4cbe-bf4d-b4303bc4e7d9 service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:08.508321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 843/1699] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:05:07 2022] DELETE /identity/v3/users/e21c2486e9a342f588402f15d692bd49 => generated 0 bytes in 648 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:05:08.540262 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 857/1700] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:08 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:09.764238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e32466e-5bc7-4c48-941f-5526d429d82f admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:09.767230 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-5e32466e-5bc7-4c48-941f-5526d429d82f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:09.831403 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8998fbc-4ddc-4ce2-9855-5806984f9b32 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:09.831863 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8998fbc-4ddc-4ce2-9855-5806984f9b32 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:09.832120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8998fbc-4ddc-4ce2-9855-5806984f9b32 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:09.834431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 858/1701] 10.222.0.42 () {54 vars in 912 bytes} [Sun Sep 4 01:05:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 01:05:09.848915 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75fe573-45f4-442f-b7fc-d326d97f9e03 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:09.849243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75fe573-45f4-442f-b7fc-d326d97f9e03 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:09.850721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75fe573-45f4-442f-b7fc-d326d97f9e03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:09.906670 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e32466e-5bc7-4c48-941f-5526d429d82f admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:09.922588 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ee0732-3b3d-4ea5-83d1-c334809278c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:09.923516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ee0732-3b3d-4ea5-83d1-c334809278c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:09.925585 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ee0732-3b3d-4ea5-83d1-c334809278c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:09.930045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ee0732-3b3d-4ea5-83d1-c334809278c8 admin admin] PATH_INFO: `/v3/projects/f14a3f986b774cb09bd5cc3f9ebeaac3` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:09.930045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ee0732-3b3d-4ea5-83d1-c334809278c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f14a3f986b774cb09bd5cc3f9ebeaac3)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:09.931571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ee0732-3b3d-4ea5-83d1-c334809278c8 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:09.971684 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-e75fe573-45f4-442f-b7fc-d326d97f9e03 None None] MFA Rules not processed for user `da4f44b86ca94adf959d9a41d915c902`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:09.977292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e75fe573-45f4-442f-b7fc-d326d97f9e03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:10.114326 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e1ee0732-3b3d-4ea5-83d1-c334809278c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f14a3f986b774cb09bd5cc3f9ebeaac3', 'request_id': 'req-e1ee0732-3b3d-4ea5-83d1-c334809278c8'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:10.127911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e1ee0732-3b3d-4ea5-83d1-c334809278c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f14a3f986b774cb09bd5cc3f9ebeaac3', 'request_id': 'req-e1ee0732-3b3d-4ea5-83d1-c334809278c8'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:10.167796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 859/1702] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 4 01:05:09 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 320 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:10.168690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75fe573-45f4-442f-b7fc-d326d97f9e03 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:10.174777 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e75fe573-45f4-442f-b7fc-d326d97f9e03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:10.200073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e1ee0732-3b3d-4ea5-83d1-c334809278c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f14a3f986b774cb09bd5cc3f9ebeaac3', 'request_id': 'req-e1ee0732-3b3d-4ea5-83d1-c334809278c8'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:10.213647 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75fe573-45f4-442f-b7fc-d326d97f9e03 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:10.220620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e1ee0732-3b3d-4ea5-83d1-c334809278c8 admin admin] The token cache is being invalidate because project f14a3f986b774cb09bd5cc3f9ebeaac3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:05:10.221283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e1ee0732-3b3d-4ea5-83d1-c334809278c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:10.222232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e1ee0732-3b3d-4ea5-83d1-c334809278c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:10.227958 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 844/1703] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:05:09 2022] DELETE /identity/v3/projects/f14a3f986b774cb09bd5cc3f9ebeaac3 => generated 0 bytes in 461 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:10.229478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ee0732-3b3d-4ea5-83d1-c334809278c8 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:10.231150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e1ee0732-3b3d-4ea5-83d1-c334809278c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:10.255347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:10.351832 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99fbe60-e210-40b0-a748-e4b6f1be397b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:10.353557 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99fbe60-e210-40b0-a748-e4b6f1be397b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:10.354237 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99fbe60-e210-40b0-a748-e4b6f1be397b admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:10.354768 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99fbe60-e210-40b0-a748-e4b6f1be397b admin admin] PATH_INFO: `/v3/users/5c092b8466ca48f8878491edb6dcb378` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:10.358502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99fbe60-e210-40b0-a748-e4b6f1be397b admin admin] RBAC: Authorizing `identity:delete_user(user_id=5c092b8466ca48f8878491edb6dcb378)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:10.364488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99fbe60-e210-40b0-a748-e4b6f1be397b admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:10.395669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ee0732-3b3d-4ea5-83d1-c334809278c8 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:10.401972 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6525b71a-3d7c-4e14-a27a-f4ca1aff5a41 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:10.403040 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6525b71a-3d7c-4e14-a27a-f4ca1aff5a41 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:10.403501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6525b71a-3d7c-4e14-a27a-f4ca1aff5a41 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:10.403924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6525b71a-3d7c-4e14-a27a-f4ca1aff5a41 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:10.409793 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6525b71a-3d7c-4e14-a27a-f4ca1aff5a41 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:10.422877 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6525b71a-3d7c-4e14-a27a-f4ca1aff5a41 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:10.424670 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6525b71a-3d7c-4e14-a27a-f4ca1aff5a41 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:10.435380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 845/1704] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:10 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:10.446790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abec263-d6c9-4c16-8e60-62f04ab549e3 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:10.451803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abec263-d6c9-4c16-8e60-62f04ab549e3 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:10.453664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abec263-d6c9-4c16-8e60-62f04ab549e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:10.521508 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-3abec263-d6c9-4c16-8e60-62f04ab549e3 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:10.525990 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-3abec263-d6c9-4c16-8e60-62f04ab549e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:10.556295 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 846/1705] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:10.566633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295ea179-a75f-4eb3-bddd-899fdd80c132 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:10.568896 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295ea179-a75f-4eb3-bddd-899fdd80c132 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:10.570478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295ea179-a75f-4eb3-bddd-899fdd80c132 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:10.577998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f99fbe60-e210-40b0-a748-e4b6f1be397b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5c092b8466ca48f8878491edb6dcb378'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:10.620378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-295ea179-a75f-4eb3-bddd-899fdd80c132 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:10.621926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-295ea179-a75f-4eb3-bddd-899fdd80c132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:10.639112 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 847/1706] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:10.647617 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295ea179-a75f-4eb3-bddd-899fdd80c132 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:10.670269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295ea179-a75f-4eb3-bddd-899fdd80c132 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:10.677363 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f99fbe60-e210-40b0-a748-e4b6f1be397b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5c092b8466ca48f8878491edb6dcb378'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:10.700085 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e60e4e-0954-4a7b-9d4c-cd0cab9eeb7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:10.700711 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e60e4e-0954-4a7b-9d4c-cd0cab9eeb7a admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:10.702163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:10.704114 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e60e4e-0954-4a7b-9d4c-cd0cab9eeb7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:10.704456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e60e4e-0954-4a7b-9d4c-cd0cab9eeb7a admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:10.705182 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e60e4e-0954-4a7b-9d4c-cd0cab9eeb7a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:05:10.705494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e60e4e-0954-4a7b-9d4c-cd0cab9eeb7a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:10.708996 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f99fbe60-e210-40b0-a748-e4b6f1be397b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5c092b8466ca48f8878491edb6dcb378'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:10.710798 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e60e4e-0954-4a7b-9d4c-cd0cab9eeb7a admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:10.713416 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f99fbe60-e210-40b0-a748-e4b6f1be397b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5c092b8466ca48f8878491edb6dcb378'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:10.759181 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f99fbe60-e210-40b0-a748-e4b6f1be397b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5c092b8466ca48f8878491edb6dcb378'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:10.792307 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 848/1707] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:05:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:05:10.813508 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e60e4e-0954-4a7b-9d4c-cd0cab9eeb7a admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:10.813778 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 860/1708] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:05:10 2022] DELETE /identity/v3/users/5c092b8466ca48f8878491edb6dcb378 => generated 0 bytes in 625 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:10.821485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-c6e60e4e-0954-4a7b-9d4c-cd0cab9eeb7a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:10.853704 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e60e4e-0954-4a7b-9d4c-cd0cab9eeb7a admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:10.966483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba4da26-18c4-4ec2-95de-5e69fc1427cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:10.968317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba4da26-18c4-4ec2-95de-5e69fc1427cb admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:10.968961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba4da26-18c4-4ec2-95de-5e69fc1427cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:10.970009 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba4da26-18c4-4ec2-95de-5e69fc1427cb admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:10.970838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba4da26-18c4-4ec2-95de-5e69fc1427cb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:10.973866 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba4da26-18c4-4ec2-95de-5e69fc1427cb admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:11.140368 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 849/1709] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:10 2022] POST /identity/v3/projects => generated 363 bytes in 328 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:11.146272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99fbe60-e210-40b0-a748-e4b6f1be397b admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:11.170349 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99fbe60-e210-40b0-a748-e4b6f1be397b admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:11.339721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1618a5f5-992a-426b-8741-dc62e01888eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:11.345527 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1618a5f5-992a-426b-8741-dc62e01888eb admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:11.345967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1618a5f5-992a-426b-8741-dc62e01888eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:11.346440 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1618a5f5-992a-426b-8741-dc62e01888eb admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:11.347268 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1618a5f5-992a-426b-8741-dc62e01888eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:11.350899 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1618a5f5-992a-426b-8741-dc62e01888eb admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:11.487659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 861/1710] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:11 2022] POST /identity/v3/users => generated 282 bytes in 343 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:11.492266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba4da26-18c4-4ec2-95de-5e69fc1427cb admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:11.522299 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba4da26-18c4-4ec2-95de-5e69fc1427cb admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:11.538451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd66c7b-8ce7-40b1-b1ce-b357da8d7918 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:11.539089 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd66c7b-8ce7-40b1-b1ce-b357da8d7918 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:11.539370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd66c7b-8ce7-40b1-b1ce-b357da8d7918 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:11.539643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd66c7b-8ce7-40b1-b1ce-b357da8d7918 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:11.540301 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd66c7b-8ce7-40b1-b1ce-b357da8d7918 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:11.541927 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd66c7b-8ce7-40b1-b1ce-b357da8d7918 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:11.581348 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 850/1711] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:11 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:05:11.585608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1618a5f5-992a-426b-8741-dc62e01888eb admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:11.651538 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1618a5f5-992a-426b-8741-dc62e01888eb admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:11.661873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4316ccdf-12d6-492b-a909-12e17f4732ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:11.663799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4316ccdf-12d6-492b-a909-12e17f4732ce admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:11.665475 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4316ccdf-12d6-492b-a909-12e17f4732ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:11.665475 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4316ccdf-12d6-492b-a909-12e17f4732ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:11.665475 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4316ccdf-12d6-492b-a909-12e17f4732ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:11.672985 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4316ccdf-12d6-492b-a909-12e17f4732ce admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:11.720518 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 862/1712] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:11 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:11.720722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd66c7b-8ce7-40b1-b1ce-b357da8d7918 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:11.789056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd66c7b-8ce7-40b1-b1ce-b357da8d7918 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:11.815819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1557ae-737d-4426-9b0b-34a3928485e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:11.822068 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1557ae-737d-4426-9b0b-34a3928485e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:11.822524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1557ae-737d-4426-9b0b-34a3928485e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:11.841226 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1557ae-737d-4426-9b0b-34a3928485e9 admin admin] PATH_INFO: `/v3/projects/e4b9f0ce5a8c47cb85cfc09bbeb8450c/users/05898be96f1c489ea62c7182a27289bb/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:11.945758 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1557ae-737d-4426-9b0b-34a3928485e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e4b9f0ce5a8c47cb85cfc09bbeb8450c, user_id=05898be96f1c489ea62c7182a27289bb, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:11.955856 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1557ae-737d-4426-9b0b-34a3928485e9 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:12.080084 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 851/1713] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:11 2022] PUT /identity/v3/projects/e4b9f0ce5a8c47cb85cfc09bbeb8450c/users/05898be96f1c489ea62c7182a27289bb/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:12.092315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f583550e-b2b3-438e-824e-e2845c520f73 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:12.092874 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f583550e-b2b3-438e-824e-e2845c520f73 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:12.093273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f583550e-b2b3-438e-824e-e2845c520f73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:12.215686 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1557ae-737d-4426-9b0b-34a3928485e9 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:12.224430 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2e1557ae-737d-4426-9b0b-34a3928485e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:12.229144 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-f583550e-b2b3-438e-824e-e2845c520f73 None None] MFA Rules not processed for user `05898be96f1c489ea62c7182a27289bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:12.234864 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-f583550e-b2b3-438e-824e-e2845c520f73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:12.376454 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:12.456856 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1557ae-737d-4426-9b0b-34a3928485e9 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:12.470618 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67af8b1-548a-4b3e-aaee-d3a8837a92f1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b29f444d56ec49308f7fef16f4d1eea5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:12.471359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67af8b1-548a-4b3e-aaee-d3a8837a92f1 service placement] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:12.471639 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67af8b1-548a-4b3e-aaee-d3a8837a92f1 service placement] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:12.473233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67af8b1-548a-4b3e-aaee-d3a8837a92f1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:12.477817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a67af8b1-548a-4b3e-aaee-d3a8837a92f1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:12.530048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67af8b1-548a-4b3e-aaee-d3a8837a92f1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:12.541327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67af8b1-548a-4b3e-aaee-d3a8837a92f1 service placement] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:12.545593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 863/1714] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:12 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 454 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:12.550419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f583550e-b2b3-438e-824e-e2845c520f73 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:12.589066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 852/1715] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:12 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:12.811208 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:12.829294 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f583550e-b2b3-438e-824e-e2845c520f73 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:12.858711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423cdf4a-3f6b-4f34-8ee8-51592eb85cd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:12.860142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423cdf4a-3f6b-4f34-8ee8-51592eb85cd4 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:12.860441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423cdf4a-3f6b-4f34-8ee8-51592eb85cd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:12.860734 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423cdf4a-3f6b-4f34-8ee8-51592eb85cd4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:12.862087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423cdf4a-3f6b-4f34-8ee8-51592eb85cd4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:12.865642 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423cdf4a-3f6b-4f34-8ee8-51592eb85cd4 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:13.001670 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 864/1716] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:12 2022] POST /identity/v3/projects => generated 365 bytes in 448 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:13.001855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67af8b1-548a-4b3e-aaee-d3a8837a92f1 service placement] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:13.019707 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67af8b1-548a-4b3e-aaee-d3a8837a92f1 service placement] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:13.100092 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719dd6b4-8690-4976-aaac-4c7f6e47a352 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:13.100588 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719dd6b4-8690-4976-aaac-4c7f6e47a352 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:13.101549 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719dd6b4-8690-4976-aaac-4c7f6e47a352 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:13.165566 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-719dd6b4-8690-4976-aaac-4c7f6e47a352 None None] MFA Rules not processed for user `44993cd93360442bb0afb28778ee83bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:13.165566 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-719dd6b4-8690-4976-aaac-4c7f6e47a352 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:13.236015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe102492-117e-4453-acbd-33a03d1a94c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:13.237268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe102492-117e-4453-acbd-33a03d1a94c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:13.237660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe102492-117e-4453-acbd-33a03d1a94c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:13.238041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe102492-117e-4453-acbd-33a03d1a94c7 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:13.238888 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe102492-117e-4453-acbd-33a03d1a94c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:13.246283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe102492-117e-4453-acbd-33a03d1a94c7 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:13.321178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 865/1717] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:13 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:05:13.321178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719dd6b4-8690-4976-aaac-4c7f6e47a352 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:13.331399 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-719dd6b4-8690-4976-aaac-4c7f6e47a352 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:13.380877 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719dd6b4-8690-4976-aaac-4c7f6e47a352 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:13.402060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:13.402803 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:13.403091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:13.403362 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da admin admin] PATH_INFO: `/v3/projects/8cd5d05953314eaab3d768e33897858f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:13.405877 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da admin admin] RBAC: Authorizing `identity:delete_project(project_id=8cd5d05953314eaab3d768e33897858f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:13.410001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:13.424054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 853/1718] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:13 2022] POST /identity/v3/users => generated 289 bytes in 422 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:13.428501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe102492-117e-4453-acbd-33a03d1a94c7 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:13.447551 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-fe102492-117e-4453-acbd-33a03d1a94c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:13.605006 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8cd5d05953314eaab3d768e33897858f', 'request_id': 'req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:13.615602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8cd5d05953314eaab3d768e33897858f', 'request_id': 'req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:13.624739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8cd5d05953314eaab3d768e33897858f', 'request_id': 'req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:13.639225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe102492-117e-4453-acbd-33a03d1a94c7 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:13.692347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da admin admin] The token cache is being invalidate because project 8cd5d05953314eaab3d768e33897858f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:05:13.693165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:13.694265 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:13.698032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 866/1719] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:05:13 2022] DELETE /identity/v3/projects/8cd5d05953314eaab3d768e33897858f => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:13.699507 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:13.702609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:13.724184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:13.745753 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4aaa9f8-96a0-42d8-823b-411b042d8ad9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:13.747804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4aaa9f8-96a0-42d8-823b-411b042d8ad9 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:13.750873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4aaa9f8-96a0-42d8-823b-411b042d8ad9 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:13.751074 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4aaa9f8-96a0-42d8-823b-411b042d8ad9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:13.757055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e4aaa9f8-96a0-42d8-823b-411b042d8ad9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:13.859644 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:13.887708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e137b48b-42e4-4c49-b7aa-bcb6fc3077da admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:13.896188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05747cea-5240-4888-bcdb-59cad6f27ee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:13.896999 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05747cea-5240-4888-bcdb-59cad6f27ee2 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:13.897412 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05747cea-5240-4888-bcdb-59cad6f27ee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:13.897808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05747cea-5240-4888-bcdb-59cad6f27ee2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:13.898691 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05747cea-5240-4888-bcdb-59cad6f27ee2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:13.900589 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05747cea-5240-4888-bcdb-59cad6f27ee2 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:13.919026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4aaa9f8-96a0-42d8-823b-411b042d8ad9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:13.919523 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 867/1720] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:13 2022] GET /identity/v3/roles => generated 1420 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:13.920664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4aaa9f8-96a0-42d8-823b-411b042d8ad9 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:13.921164 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05747cea-5240-4888-bcdb-59cad6f27ee2 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:13.923446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-05747cea-5240-4888-bcdb-59cad6f27ee2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:13.936877 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 854/1721] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:13 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 513 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:13.937713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4aaa9f8-96a0-42d8-823b-411b042d8ad9 service nova] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:13.941340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e4aaa9f8-96a0-42d8-823b-411b042d8ad9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:13.959836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05747cea-5240-4888-bcdb-59cad6f27ee2 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:13.976750 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4aaa9f8-96a0-42d8-823b-411b042d8ad9 service nova] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:13.982269 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151e31ba-de2b-49c6-988d-2733dc9b601b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:13.985185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151e31ba-de2b-49c6-988d-2733dc9b601b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:13.985486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151e31ba-de2b-49c6-988d-2733dc9b601b admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:13.989998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151e31ba-de2b-49c6-988d-2733dc9b601b admin admin] PATH_INFO: `/v3/users/16e735fec04b41839258a8fd87ef7b2b` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:13.990475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b586a-ee2a-45ab-ab0d-eb7bf400a2f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:13.991288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b586a-ee2a-45ab-ab0d-eb7bf400a2f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:13.991401 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151e31ba-de2b-49c6-988d-2733dc9b601b admin admin] RBAC: Authorizing `identity:delete_user(user_id=16e735fec04b41839258a8fd87ef7b2b)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:13.991576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b586a-ee2a-45ab-ab0d-eb7bf400a2f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:13.991847 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b586a-ee2a-45ab-ab0d-eb7bf400a2f0 admin admin] PATH_INFO: `/v3/users/95a5add16c664d53b6c65f9469212093` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:13.998257 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6b586a-ee2a-45ab-ab0d-eb7bf400a2f0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=95a5add16c664d53b6c65f9469212093)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:14.002381 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151e31ba-de2b-49c6-988d-2733dc9b601b admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:14.002497 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6b586a-ee2a-45ab-ab0d-eb7bf400a2f0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:14.248037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5b6b586a-ee2a-45ab-ab0d-eb7bf400a2f0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95a5add16c664d53b6c65f9469212093'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:14.251855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5b6b586a-ee2a-45ab-ab0d-eb7bf400a2f0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95a5add16c664d53b6c65f9469212093'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:14.305493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-151e31ba-de2b-49c6-988d-2733dc9b601b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '16e735fec04b41839258a8fd87ef7b2b'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:14.353155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5b6b586a-ee2a-45ab-ab0d-eb7bf400a2f0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95a5add16c664d53b6c65f9469212093'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:14.390457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5b6b586a-ee2a-45ab-ab0d-eb7bf400a2f0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95a5add16c664d53b6c65f9469212093'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:14.391321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5b6b586a-ee2a-45ab-ab0d-eb7bf400a2f0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95a5add16c664d53b6c65f9469212093'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:14.444230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-151e31ba-de2b-49c6-988d-2733dc9b601b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '16e735fec04b41839258a8fd87ef7b2b'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:14.459840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 855/1722] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:05:13 2022] DELETE /identity/v3/users/95a5add16c664d53b6c65f9469212093 => generated 0 bytes in 522 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:14.468430 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b586a-ee2a-45ab-ab0d-eb7bf400a2f0 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:14.473808 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-5b6b586a-ee2a-45ab-ab0d-eb7bf400a2f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:14.475252 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-151e31ba-de2b-49c6-988d-2733dc9b601b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '16e735fec04b41839258a8fd87ef7b2b'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:14.477909 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-151e31ba-de2b-49c6-988d-2733dc9b601b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '16e735fec04b41839258a8fd87ef7b2b'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:14.526963 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-151e31ba-de2b-49c6-988d-2733dc9b601b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '16e735fec04b41839258a8fd87ef7b2b'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:14.556431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 868/1723] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:05:13 2022] DELETE /identity/v3/users/16e735fec04b41839258a8fd87ef7b2b => generated 0 bytes in 636 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:14.557924 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f870bb-dbe8-4bd7-bb1b-156f7cb3de70 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:14.558213 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f870bb-dbe8-4bd7-bb1b-156f7cb3de70 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:14.564565 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f870bb-dbe8-4bd7-bb1b-156f7cb3de70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:14.681138 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-f9f870bb-dbe8-4bd7-bb1b-156f7cb3de70 None None] MFA Rules not processed for user `fbcd4c19e766446188e3b3f41a44e2e4`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:14.694913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-f9f870bb-dbe8-4bd7-bb1b-156f7cb3de70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:14.695073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b586a-ee2a-45ab-ab0d-eb7bf400a2f0 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:14.855921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:14.878974 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 869/1724] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:14 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 322 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:14.886870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a4cf30-1128-4c1e-9f62-7cdab2c4d11c None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:14.886870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a4cf30-1128-4c1e-9f62-7cdab2c4d11c None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:14.886870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a4cf30-1128-4c1e-9f62-7cdab2c4d11c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:14.891384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfdebaa-b31d-4cf7-9ede-c466ef12f8ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:14.892524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfdebaa-b31d-4cf7-9ede-c466ef12f8ea admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:14.893126 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfdebaa-b31d-4cf7-9ede-c466ef12f8ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:14.896175 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfdebaa-b31d-4cf7-9ede-c466ef12f8ea admin admin] PATH_INFO: `/v3/projects/59e345a392104034b113adab2fa35c8f/users/8d501ad64e734d8b9e5378a17eef9f42/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:14.930278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-b0a4cf30-1128-4c1e-9f62-7cdab2c4d11c None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:14.931523 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b0a4cf30-1128-4c1e-9f62-7cdab2c4d11c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:14.936044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfdebaa-b31d-4cf7-9ede-c466ef12f8ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=59e345a392104034b113adab2fa35c8f, user_id=8d501ad64e734d8b9e5378a17eef9f42, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:14.939171 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfdebaa-b31d-4cf7-9ede-c466ef12f8ea admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:14.954794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 870/1725] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:14.958293 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45860b1-7c49-4f78-84e8-aa9cbf5e5360 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:14.958812 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45860b1-7c49-4f78-84e8-aa9cbf5e5360 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:14.959230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45860b1-7c49-4f78-84e8-aa9cbf5e5360 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:15.012877 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-e45860b1-7c49-4f78-84e8-aa9cbf5e5360 None None] MFA Rules not processed for user `f5e4723258c64ce5885154dded2d3d43`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:15.014371 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e45860b1-7c49-4f78-84e8-aa9cbf5e5360 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:15.097555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 856/1726] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:14 2022] PUT /identity/v3/projects/59e345a392104034b113adab2fa35c8f/users/8d501ad64e734d8b9e5378a17eef9f42/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 636 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:15.103650 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfdebaa-b31d-4cf7-9ede-c466ef12f8ea admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:15.116236 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ddfdebaa-b31d-4cf7-9ede-c466ef12f8ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:15.238004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 871/1727] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:14 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:15.242461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45860b1-7c49-4f78-84e8-aa9cbf5e5360 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:15.243355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e45860b1-7c49-4f78-84e8-aa9cbf5e5360 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:15.322542 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfdebaa-b31d-4cf7-9ede-c466ef12f8ea admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:15.336534 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a44e95f-0a2a-4656-b527-c60166fdd46b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:15.337204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a44e95f-0a2a-4656-b527-c60166fdd46b service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:15.337489 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a44e95f-0a2a-4656-b527-c60166fdd46b service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:15.337764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a44e95f-0a2a-4656-b527-c60166fdd46b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:15.397782 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45860b1-7c49-4f78-84e8-aa9cbf5e5360 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:15.509055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a44e95f-0a2a-4656-b527-c60166fdd46b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:15.513208 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a44e95f-0a2a-4656-b527-c60166fdd46b service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:15.515914 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc7ecf2-9f54-414b-a225-d262452e470f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:15.517565 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc7ecf2-9f54-414b-a225-d262452e470f service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:15.517859 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc7ecf2-9f54-414b-a225-d262452e470f service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:15.518145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc7ecf2-9f54-414b-a225-d262452e470f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:15.539883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 857/1728] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:15 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:15.559013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a44e95f-0a2a-4656-b527-c60166fdd46b service nova] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:15.569467 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6a44e95f-0a2a-4656-b527-c60166fdd46b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:15.661563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc7ecf2-9f54-414b-a225-d262452e470f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:15.663041 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc7ecf2-9f54-414b-a225-d262452e470f service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:15.678760 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:15.691179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 872/1729] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:15.692774 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc7ecf2-9f54-414b-a225-d262452e470f service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:15.695986 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-cdc7ecf2-9f54-414b-a225-d262452e470f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:15.724377 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc7ecf2-9f54-414b-a225-d262452e470f service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:15.734269 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3081f60a-1621-4332-b110-c1d5c005a178 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:15.734926 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3081f60a-1621-4332-b110-c1d5c005a178 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:15.735198 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3081f60a-1621-4332-b110-c1d5c005a178 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:15.735468 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3081f60a-1621-4332-b110-c1d5c005a178 admin admin] PATH_INFO: `/v3/projects/d19b79b17abb4d78a4cc302a316f3688` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:15.737823 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3081f60a-1621-4332-b110-c1d5c005a178 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d19b79b17abb4d78a4cc302a316f3688)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:15.740632 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3081f60a-1621-4332-b110-c1d5c005a178 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:15.812337 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a44e95f-0a2a-4656-b527-c60166fdd46b service nova] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:15.822834 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5475284-37ad-4719-9bc0-676783ee51c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:15.824271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5475284-37ad-4719-9bc0-676783ee51c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:15.825331 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5475284-37ad-4719-9bc0-676783ee51c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:15.826384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5475284-37ad-4719-9bc0-676783ee51c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:15.832606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5475284-37ad-4719-9bc0-676783ee51c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:15.834838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5475284-37ad-4719-9bc0-676783ee51c8 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:15.873560 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 858/1730] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:15 2022] GET /identity/v3/roles => generated 1420 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:15.874615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5475284-37ad-4719-9bc0-676783ee51c8 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:15.879749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f5475284-37ad-4719-9bc0-676783ee51c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:15.918927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3081f60a-1621-4332-b110-c1d5c005a178 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd19b79b17abb4d78a4cc302a316f3688', 'request_id': 'req-3081f60a-1621-4332-b110-c1d5c005a178'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:15.926717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3081f60a-1621-4332-b110-c1d5c005a178 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd19b79b17abb4d78a4cc302a316f3688', 'request_id': 'req-3081f60a-1621-4332-b110-c1d5c005a178'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:15.926989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5475284-37ad-4719-9bc0-676783ee51c8 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:15.944440 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3081f60a-1621-4332-b110-c1d5c005a178 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd19b79b17abb4d78a4cc302a316f3688', 'request_id': 'req-3081f60a-1621-4332-b110-c1d5c005a178'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:16.057507 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3081f60a-1621-4332-b110-c1d5c005a178 admin admin] The token cache is being invalidate because project d19b79b17abb4d78a4cc302a316f3688 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:05:16.061741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3081f60a-1621-4332-b110-c1d5c005a178 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:16.061741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3081f60a-1621-4332-b110-c1d5c005a178 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:16.069279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 873/1731] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:05:15 2022] DELETE /identity/v3/projects/d19b79b17abb4d78a4cc302a316f3688 => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:16.072140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3081f60a-1621-4332-b110-c1d5c005a178 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:16.077052 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-3081f60a-1621-4332-b110-c1d5c005a178 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:16.149023 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48d9a44-2469-41b3-82c2-bf0c5fbceef0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:16.149700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d9a44-2469-41b3-82c2-bf0c5fbceef0 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:16.150046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d9a44-2469-41b3-82c2-bf0c5fbceef0 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:16.150444 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d9a44-2469-41b3-82c2-bf0c5fbceef0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:16.164201 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b48d9a44-2469-41b3-82c2-bf0c5fbceef0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:16.249504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3081f60a-1621-4332-b110-c1d5c005a178 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:16.260085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96176854-fb8b-4f41-b65b-98ce9b8edcb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:16.260796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96176854-fb8b-4f41-b65b-98ce9b8edcb6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:16.261118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96176854-fb8b-4f41-b65b-98ce9b8edcb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:16.265610 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96176854-fb8b-4f41-b65b-98ce9b8edcb6 admin admin] PATH_INFO: `/v3/projects/59e345a392104034b113adab2fa35c8f/users/8d501ad64e734d8b9e5378a17eef9f42/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:16.274487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96176854-fb8b-4f41-b65b-98ce9b8edcb6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=59e345a392104034b113adab2fa35c8f, user_id=8d501ad64e734d8b9e5378a17eef9f42, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:16.277253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96176854-fb8b-4f41-b65b-98ce9b8edcb6 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:16.277903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48d9a44-2469-41b3-82c2-bf0c5fbceef0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:16.280349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48d9a44-2469-41b3-82c2-bf0c5fbceef0 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:16.316203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 859/1732] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:15 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:16.320023 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922468c3-ab77-4dbf-bf41-ae16efabc56b None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:16.324019 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922468c3-ab77-4dbf-bf41-ae16efabc56b None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:16.324827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922468c3-ab77-4dbf-bf41-ae16efabc56b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:16.330114 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 874/1733] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:16 2022] PUT /identity/v3/projects/59e345a392104034b113adab2fa35c8f/users/8d501ad64e734d8b9e5378a17eef9f42/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:16.335369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897d74f9-7844-4eda-9cab-b6769911ec7d None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:16.335659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897d74f9-7844-4eda-9cab-b6769911ec7d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:16.335942 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897d74f9-7844-4eda-9cab-b6769911ec7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:16.379220 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-922468c3-ab77-4dbf-bf41-ae16efabc56b None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:16.384180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-922468c3-ab77-4dbf-bf41-ae16efabc56b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:16.409615 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-897d74f9-7844-4eda-9cab-b6769911ec7d None None] MFA Rules not processed for user `8d501ad64e734d8b9e5378a17eef9f42`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:16.415724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-897d74f9-7844-4eda-9cab-b6769911ec7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:16.546240 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 860/1734] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:16.547946 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3978ee28-5098-4211-8ee2-b2d7b86990f5 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:16.548240 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3978ee28-5098-4211-8ee2-b2d7b86990f5 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:16.548533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3978ee28-5098-4211-8ee2-b2d7b86990f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:16.581744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 875/1735] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:16 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:16.583286 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-3978ee28-5098-4211-8ee2-b2d7b86990f5 None None] MFA Rules not processed for user `bd13df3a2d0e4182bbd86dab7035c371`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:16.584512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-3978ee28-5098-4211-8ee2-b2d7b86990f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:16.585195 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b681ff54-b021-4908-98fd-1cb678513778 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:16.585687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b681ff54-b021-4908-98fd-1cb678513778 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:16.586167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b681ff54-b021-4908-98fd-1cb678513778 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:16.616691 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-b681ff54-b021-4908-98fd-1cb678513778 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:16.618869 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b681ff54-b021-4908-98fd-1cb678513778 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:16.636395 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 876/1736] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:16.637862 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b681ff54-b021-4908-98fd-1cb678513778 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:16.640062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b681ff54-b021-4908-98fd-1cb678513778 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:16.676534 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 861/1737] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:16 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:16.676534 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3978ee28-5098-4211-8ee2-b2d7b86990f5 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:16.686888 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3978ee28-5098-4211-8ee2-b2d7b86990f5 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:16.693117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13650da5-ce65-4d4d-9ce3-0265b88474f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:16.693739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13650da5-ce65-4d4d-9ce3-0265b88474f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:16.694023 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13650da5-ce65-4d4d-9ce3-0265b88474f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:16.694291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13650da5-ce65-4d4d-9ce3-0265b88474f1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:16.694951 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13650da5-ce65-4d4d-9ce3-0265b88474f1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:05:16.695258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13650da5-ce65-4d4d-9ce3-0265b88474f1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:16.696869 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13650da5-ce65-4d4d-9ce3-0265b88474f1 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:16.707742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b681ff54-b021-4908-98fd-1cb678513778 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:16.711489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24db429-d44b-4902-a5cd-ec8592ea7396 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:16.712107 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24db429-d44b-4902-a5cd-ec8592ea7396 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:16.712385 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24db429-d44b-4902-a5cd-ec8592ea7396 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:16.712657 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24db429-d44b-4902-a5cd-ec8592ea7396 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:16.714444 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 862/1738] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:05:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:05:16.715858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93bac9b-ea7a-4018-b0ca-be95beed35db None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:16.716150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93bac9b-ea7a-4018-b0ca-be95beed35db None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:16.716424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93bac9b-ea7a-4018-b0ca-be95beed35db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:16.725639 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24db429-d44b-4902-a5cd-ec8592ea7396 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:16.727080 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24db429-d44b-4902-a5cd-ec8592ea7396 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:16.737356 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 877/1739] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:16 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:16.737515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-e93bac9b-ea7a-4018-b0ca-be95beed35db None None] MFA Rules not processed for user `bd13df3a2d0e4182bbd86dab7035c371`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:16.737997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24db429-d44b-4902-a5cd-ec8592ea7396 service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:16.738765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e93bac9b-ea7a-4018-b0ca-be95beed35db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:16.748295 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24db429-d44b-4902-a5cd-ec8592ea7396 service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:16.749516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 863/1740] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:16 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:16.758608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a30916-9f00-45d8-8528-79fb50745250 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:16.759143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a30916-9f00-45d8-8528-79fb50745250 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:16.759436 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a30916-9f00-45d8-8528-79fb50745250 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:16.759698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a30916-9f00-45d8-8528-79fb50745250 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:16.760343 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a30916-9f00-45d8-8528-79fb50745250 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:16.762417 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a30916-9f00-45d8-8528-79fb50745250 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:16.766035 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93bac9b-ea7a-4018-b0ca-be95beed35db None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:16.767653 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e93bac9b-ea7a-4018-b0ca-be95beed35db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:16.854614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93bac9b-ea7a-4018-b0ca-be95beed35db None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:16.860586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf19768-93ab-448e-8856-34e87b4a175f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:16.861506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf19768-93ab-448e-8856-34e87b4a175f service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:16.861786 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf19768-93ab-448e-8856-34e87b4a175f service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:16.862060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf19768-93ab-448e-8856-34e87b4a175f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:16.870847 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf19768-93ab-448e-8856-34e87b4a175f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:16.872504 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf19768-93ab-448e-8856-34e87b4a175f service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:16.882803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 864/1741] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:16 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:17.300130 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016c3653-a920-4a7b-9075-d4fd6a78fed8 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:17.301275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016c3653-a920-4a7b-9075-d4fd6a78fed8 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:17.301275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016c3653-a920-4a7b-9075-d4fd6a78fed8 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:17.302738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 865/1742] 10.222.0.42 () {54 vars in 912 bytes} [Sun Sep 4 01:05:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 01:05:17.307910 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de41ee02-7c4d-4f14-9a3d-a7098597eff3 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:17.308283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de41ee02-7c4d-4f14-9a3d-a7098597eff3 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:17.308895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de41ee02-7c4d-4f14-9a3d-a7098597eff3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:17.331785 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-de41ee02-7c4d-4f14-9a3d-a7098597eff3 None None] MFA Rules not processed for user `da4f44b86ca94adf959d9a41d915c902`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:17.333320 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-de41ee02-7c4d-4f14-9a3d-a7098597eff3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:17.384968 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 866/1743] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 4 01:05:17 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:17.396666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de41ee02-7c4d-4f14-9a3d-a7098597eff3 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:17.398051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-de41ee02-7c4d-4f14-9a3d-a7098597eff3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:17.408422 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de41ee02-7c4d-4f14-9a3d-a7098597eff3 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:17.413332 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c428e18-1371-4925-ab1f-22c478e81cbe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:17.413974 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c428e18-1371-4925-ab1f-22c478e81cbe service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:17.414254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c428e18-1371-4925-ab1f-22c478e81cbe service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:17.414532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c428e18-1371-4925-ab1f-22c478e81cbe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:17.422624 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c428e18-1371-4925-ab1f-22c478e81cbe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:17.424098 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c428e18-1371-4925-ab1f-22c478e81cbe service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:17.433895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 867/1744] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:17 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:05:17.818178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 878/1745] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:16 2022] POST /identity/v3/projects => generated 369 bytes in 1080 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:17.827221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c428e18-1371-4925-ab1f-22c478e81cbe service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:17.859813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c428e18-1371-4925-ab1f-22c478e81cbe service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:17.918272 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:18.101689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825b9745-ed94-478a-ab18-8fda8eb473a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:18.102488 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825b9745-ed94-478a-ab18-8fda8eb473a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:18.103452 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825b9745-ed94-478a-ab18-8fda8eb473a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:18.103681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825b9745-ed94-478a-ab18-8fda8eb473a8 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:18.104423 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825b9745-ed94-478a-ab18-8fda8eb473a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:18.106873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825b9745-ed94-478a-ab18-8fda8eb473a8 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:18.192315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e24488a-b78b-4d31-b621-920f517b0579 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:18.192676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e24488a-b78b-4d31-b621-920f517b0579 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:18.192900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e24488a-b78b-4d31-b621-920f517b0579 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:18.260915 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-3e24488a-b78b-4d31-b621-920f517b0579 None None] MFA Rules not processed for user `bd13df3a2d0e4182bbd86dab7035c371`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:18.269417 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-3e24488a-b78b-4d31-b621-920f517b0579 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:18.316454 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 868/1746] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:17 2022] POST /identity/v3/users => generated 285 bytes in 490 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:18.318979 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825b9745-ed94-478a-ab18-8fda8eb473a8 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:18.351630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825b9745-ed94-478a-ab18-8fda8eb473a8 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:18.365065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c384e1-7d69-4bec-959a-55acb7501f5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:18.369505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c384e1-7d69-4bec-959a-55acb7501f5f admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:18.369505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c384e1-7d69-4bec-959a-55acb7501f5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:18.369505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c384e1-7d69-4bec-959a-55acb7501f5f admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:18.385491 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c384e1-7d69-4bec-959a-55acb7501f5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:18.391873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c384e1-7d69-4bec-959a-55acb7501f5f admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:18.453231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 869/1747] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:18 2022] GET /identity/v3/roles => generated 1420 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:18.458136 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c384e1-7d69-4bec-959a-55acb7501f5f admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:18.494268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c384e1-7d69-4bec-959a-55acb7501f5f admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:18.513949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be54075d-d8de-476b-bb29-0a67074700fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:18.516562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be54075d-d8de-476b-bb29-0a67074700fe admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:18.517950 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be54075d-d8de-476b-bb29-0a67074700fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:18.518351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be54075d-d8de-476b-bb29-0a67074700fe admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:18.521679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be54075d-d8de-476b-bb29-0a67074700fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:18.530102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be54075d-d8de-476b-bb29-0a67074700fe admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:18.530529 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 879/1748] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:18 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 335 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:05:18.639368 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e24488a-b78b-4d31-b621-920f517b0579 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:18.639556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 870/1749] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:18 2022] GET /identity/v3/roles => generated 1420 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:18.670363 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e24488a-b78b-4d31-b621-920f517b0579 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:18.684977 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7e0ff7-7384-47a1-897a-8e5b7d9fa45f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:18.688809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7e0ff7-7384-47a1-897a-8e5b7d9fa45f admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:18.688809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7e0ff7-7384-47a1-897a-8e5b7d9fa45f admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:18.688809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7e0ff7-7384-47a1-897a-8e5b7d9fa45f admin admin] PATH_INFO: `/v3/projects/955511478fc2489e9f057dbf13bf3fc2/users/8bf3e8f154574a648bfb56e6882d617a/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:18.731593 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be54075d-d8de-476b-bb29-0a67074700fe admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:18.737158 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-be54075d-d8de-476b-bb29-0a67074700fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:18.747672 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7e0ff7-7384-47a1-897a-8e5b7d9fa45f admin admin] RBAC: Authorizing `identity:create_grant(project_id=955511478fc2489e9f057dbf13bf3fc2, user_id=8bf3e8f154574a648bfb56e6882d617a, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:18.750508 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7e0ff7-7384-47a1-897a-8e5b7d9fa45f admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:18.760856 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be54075d-d8de-476b-bb29-0a67074700fe admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:18.787936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e559978e-daf8-433a-b188-7ee926409432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:18.788942 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e559978e-daf8-433a-b188-7ee926409432 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:18.789361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e559978e-daf8-433a-b188-7ee926409432 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:18.789740 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e559978e-daf8-433a-b188-7ee926409432 admin admin] PATH_INFO: `/v3/projects/2b4087f305b5468b973de4fe953bee24` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:18.791243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e559978e-daf8-433a-b188-7ee926409432 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2b4087f305b5468b973de4fe953bee24)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:18.793633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e559978e-daf8-433a-b188-7ee926409432 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:18.885995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 880/1750] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:18 2022] PUT /identity/v3/projects/955511478fc2489e9f057dbf13bf3fc2/users/8bf3e8f154574a648bfb56e6882d617a/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:18.887486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2187bfc-2d86-4da2-9946-5e02c7c36f8d None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:18.887486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2187bfc-2d86-4da2-9946-5e02c7c36f8d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:18.887905 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2187bfc-2d86-4da2-9946-5e02c7c36f8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:18.979161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-a2187bfc-2d86-4da2-9946-5e02c7c36f8d None None] MFA Rules not processed for user `8bf3e8f154574a648bfb56e6882d617a`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:18.980652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a2187bfc-2d86-4da2-9946-5e02c7c36f8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:19.072955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e559978e-daf8-433a-b188-7ee926409432 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2b4087f305b5468b973de4fe953bee24', 'request_id': 'req-e559978e-daf8-433a-b188-7ee926409432'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:19.080918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e559978e-daf8-433a-b188-7ee926409432 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2b4087f305b5468b973de4fe953bee24', 'request_id': 'req-e559978e-daf8-433a-b188-7ee926409432'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:19.194446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 881/1751] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:18 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:19.194446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2187bfc-2d86-4da2-9946-5e02c7c36f8d None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:19.222125 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e559978e-daf8-433a-b188-7ee926409432 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2b4087f305b5468b973de4fe953bee24', 'request_id': 'req-e559978e-daf8-433a-b188-7ee926409432'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:19.273268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e559978e-daf8-433a-b188-7ee926409432 admin admin] The token cache is being invalidate because project 2b4087f305b5468b973de4fe953bee24 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:05:19.282385 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e559978e-daf8-433a-b188-7ee926409432 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:19.283825 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e559978e-daf8-433a-b188-7ee926409432 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:19.299328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 871/1752] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:05:18 2022] DELETE /identity/v3/projects/2b4087f305b5468b973de4fe953bee24 => generated 0 bytes in 564 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:05:19.305406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e559978e-daf8-433a-b188-7ee926409432 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:19.312128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e559978e-daf8-433a-b188-7ee926409432 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:19.402874 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2187bfc-2d86-4da2-9946-5e02c7c36f8d None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:19.477617 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e559978e-daf8-433a-b188-7ee926409432 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:19.502993 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64cee02c-b760-47f1-8ab2-75254bf5b904 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:19.505112 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64cee02c-b760-47f1-8ab2-75254bf5b904 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:19.509669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64cee02c-b760-47f1-8ab2-75254bf5b904 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:19.510183 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64cee02c-b760-47f1-8ab2-75254bf5b904 admin admin] PATH_INFO: `/v3/users/2fb5f8174ff345519dbbf97c156972dc` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:19.517441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64cee02c-b760-47f1-8ab2-75254bf5b904 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2fb5f8174ff345519dbbf97c156972dc)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:19.521229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64cee02c-b760-47f1-8ab2-75254bf5b904 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:19.541982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:19.624094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edffc25-8933-4cd0-9409-6364941c135f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:19.624094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edffc25-8933-4cd0-9409-6364941c135f admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:19.624735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edffc25-8933-4cd0-9409-6364941c135f admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:19.625494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edffc25-8933-4cd0-9409-6364941c135f admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:19.626389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edffc25-8933-4cd0-9409-6364941c135f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:19.628643 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edffc25-8933-4cd0-9409-6364941c135f admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:19.774956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:19.786933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 882/1753] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:19 2022] POST /identity/v3/projects => generated 369 bytes in 596 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:19.788465 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edffc25-8933-4cd0-9409-6364941c135f admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:19.798711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-3edffc25-8933-4cd0-9409-6364941c135f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:19.932126 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-64cee02c-b760-47f1-8ab2-75254bf5b904 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2fb5f8174ff345519dbbf97c156972dc'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:19.933949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-64cee02c-b760-47f1-8ab2-75254bf5b904 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2fb5f8174ff345519dbbf97c156972dc'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:19.953299 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edffc25-8933-4cd0-9409-6364941c135f admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:19.965005 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3df6ec-6528-450e-8c76-cb24336ee568 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:19.968069 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3df6ec-6528-450e-8c76-cb24336ee568 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:19.968488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3df6ec-6528-450e-8c76-cb24336ee568 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:19.968887 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3df6ec-6528-450e-8c76-cb24336ee568 admin admin] PATH_INFO: `/v3/users/05898be96f1c489ea62c7182a27289bb` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:20.005526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3df6ec-6528-450e-8c76-cb24336ee568 admin admin] RBAC: Authorizing `identity:delete_user(user_id=05898be96f1c489ea62c7182a27289bb)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:20.005526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3df6ec-6528-450e-8c76-cb24336ee568 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:20.061786 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-64cee02c-b760-47f1-8ab2-75254bf5b904 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2fb5f8174ff345519dbbf97c156972dc'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:20.084490 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-64cee02c-b760-47f1-8ab2-75254bf5b904 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2fb5f8174ff345519dbbf97c156972dc'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:20.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-64cee02c-b760-47f1-8ab2-75254bf5b904 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2fb5f8174ff345519dbbf97c156972dc'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:20.119225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 872/1754] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:05:19 2022] DELETE /identity/v3/users/2fb5f8174ff345519dbbf97c156972dc => generated 0 bytes in 819 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:20.130733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64cee02c-b760-47f1-8ab2-75254bf5b904 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:20.132375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-64cee02c-b760-47f1-8ab2-75254bf5b904 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:20.252405 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3e3df6ec-6528-450e-8c76-cb24336ee568 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05898be96f1c489ea62c7182a27289bb'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:20.320391 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64cee02c-b760-47f1-8ab2-75254bf5b904 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:20.330955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37b6ca8-2efe-445f-8745-4b315f782265 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:20.338123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37b6ca8-2efe-445f-8745-4b315f782265 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:20.338576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37b6ca8-2efe-445f-8745-4b315f782265 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:20.339009 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37b6ca8-2efe-445f-8745-4b315f782265 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:20.339904 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37b6ca8-2efe-445f-8745-4b315f782265 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:20.343545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37b6ca8-2efe-445f-8745-4b315f782265 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:20.383521 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3e3df6ec-6528-450e-8c76-cb24336ee568 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05898be96f1c489ea62c7182a27289bb'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:20.460534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3e3df6ec-6528-450e-8c76-cb24336ee568 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05898be96f1c489ea62c7182a27289bb'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:20.462019 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3e3df6ec-6528-450e-8c76-cb24336ee568 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05898be96f1c489ea62c7182a27289bb'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:20.496398 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3e3df6ec-6528-450e-8c76-cb24336ee568 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05898be96f1c489ea62c7182a27289bb'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:20.500743 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 873/1755] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:20 2022] POST /identity/v3/users => generated 291 bytes in 381 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:20.502638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71a17aa-a923-46c4-ac24-54957a8f01f1 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:20.502638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71a17aa-a923-46c4-ac24-54957a8f01f1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:20.502806 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71a17aa-a923-46c4-ac24-54957a8f01f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:20.518177 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 883/1756] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:05:19 2022] DELETE /identity/v3/users/05898be96f1c489ea62c7182a27289bb => generated 0 bytes in 731 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:20.519892 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3df6ec-6528-450e-8c76-cb24336ee568 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:20.526715 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-3e3df6ec-6528-450e-8c76-cb24336ee568 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:20.724839 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:20.739000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-b71a17aa-a923-46c4-ac24-54957a8f01f1 None None] MFA Rules not processed for user `70cef7c271e046a3b49230057cc2d28f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:20.740485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b71a17aa-a923-46c4-ac24-54957a8f01f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:20.777118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3df6ec-6528-450e-8c76-cb24336ee568 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:20.801540 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd92664e-9d52-4d49-aea4-30985523c6f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:20.802600 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd92664e-9d52-4d49-aea4-30985523c6f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:20.807101 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd92664e-9d52-4d49-aea4-30985523c6f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:20.827854 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd92664e-9d52-4d49-aea4-30985523c6f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:20.830283 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd92664e-9d52-4d49-aea4-30985523c6f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:20.833291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd92664e-9d52-4d49-aea4-30985523c6f4 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:20.880065 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 884/1757] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:20 2022] GET /identity/v3/roles => generated 1420 bytes in 361 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:20.884499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1442fee-3247-495a-9777-64477067b290 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:20.886152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1442fee-3247-495a-9777-64477067b290 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:20.886152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1442fee-3247-495a-9777-64477067b290 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:20.947145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-f1442fee-3247-495a-9777-64477067b290 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:20.948422 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-f1442fee-3247-495a-9777-64477067b290 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:20.973217 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 885/1758] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:20.983915 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1442fee-3247-495a-9777-64477067b290 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:21.038231 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1442fee-3247-495a-9777-64477067b290 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:21.050072 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2f4411-ade3-4eb6-9914-65751cc769c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:21.051473 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2f4411-ade3-4eb6-9914-65751cc769c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:21.052502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2f4411-ade3-4eb6-9914-65751cc769c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:21.052843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2f4411-ade3-4eb6-9914-65751cc769c4 admin admin] PATH_INFO: `/v3/projects/d73d808e831f4d1a982b7f538a5853de/users/d46cd6214aa34547855cec69962f0cfb/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:21.067443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 874/1759] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 4 01:05:20 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 566 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:21.068999 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71a17aa-a923-46c4-ac24-54957a8f01f1 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:21.071942 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b71a17aa-a923-46c4-ac24-54957a8f01f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:21.100577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2f4411-ade3-4eb6-9914-65751cc769c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d73d808e831f4d1a982b7f538a5853de, user_id=d46cd6214aa34547855cec69962f0cfb, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:21.118448 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2f4411-ade3-4eb6-9914-65751cc769c4 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:21.223353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71a17aa-a923-46c4-ac24-54957a8f01f1 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:21.236079 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8c25c4-2c90-4edb-8e82-e43b81a5557c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:21.236991 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8c25c4-2c90-4edb-8e82-e43b81a5557c service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:21.237303 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8c25c4-2c90-4edb-8e82-e43b81a5557c service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:21.237580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8c25c4-2c90-4edb-8e82-e43b81a5557c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:21.242536 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 886/1760] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:20 2022] PUT /identity/v3/projects/d73d808e831f4d1a982b7f538a5853de/users/d46cd6214aa34547855cec69962f0cfb/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:21.242536 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2f4411-ade3-4eb6-9914-65751cc769c4 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:21.252015 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-fc2f4411-ade3-4eb6-9914-65751cc769c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:21.257460 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8c25c4-2c90-4edb-8e82-e43b81a5557c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:21.258208 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8c25c4-2c90-4edb-8e82-e43b81a5557c service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:21.288066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 875/1761] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:21.289008 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8c25c4-2c90-4edb-8e82-e43b81a5557c service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:21.309156 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8c25c4-2c90-4edb-8e82-e43b81a5557c service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:21.331224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:DELETE=5,keystone:SELECT=93,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:21.418041 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2f4411-ade3-4eb6-9914-65751cc769c4 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:21.423893 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3589f18-2344-4415-aba1-a66fb52c25c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:21.423893 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3589f18-2344-4415-aba1-a66fb52c25c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:21.423893 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3589f18-2344-4415-aba1-a66fb52c25c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:21.423893 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3589f18-2344-4415-aba1-a66fb52c25c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:21.424727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3589f18-2344-4415-aba1-a66fb52c25c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:21.424727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3589f18-2344-4415-aba1-a66fb52c25c2 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:21.430149 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:21.431267 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:21.432245 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:21.432245 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533 admin admin] PATH_INFO: `/v3/projects/42de058800864cb7b2e062ad4bd1fd46` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:21.434469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533 admin admin] RBAC: Authorizing `identity:delete_project(project_id=42de058800864cb7b2e062ad4bd1fd46)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:21.435783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 876/1762] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:21 2022] GET /identity/v3/roles => generated 1420 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:21.442636 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3589f18-2344-4415-aba1-a66fb52c25c2 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:21.445305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:21.462573 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3589f18-2344-4415-aba1-a66fb52c25c2 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:21.483420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec0f06c-9e18-4f30-944d-819fa1c284bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:21.484256 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec0f06c-9e18-4f30-944d-819fa1c284bc admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:21.485015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec0f06c-9e18-4f30-944d-819fa1c284bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:21.485392 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec0f06c-9e18-4f30-944d-819fa1c284bc admin admin] PATH_INFO: `/v3/projects/d73d808e831f4d1a982b7f538a5853de/users/d46cd6214aa34547855cec69962f0cfb/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:21.487899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec0f06c-9e18-4f30-944d-819fa1c284bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=d73d808e831f4d1a982b7f538a5853de, user_id=d46cd6214aa34547855cec69962f0cfb, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:21.492375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec0f06c-9e18-4f30-944d-819fa1c284bc admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:21.534334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 877/1763] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:21 2022] PUT /identity/v3/projects/d73d808e831f4d1a982b7f538a5853de/users/d46cd6214aa34547855cec69962f0cfb/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:21.544606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daebf61f-b239-4695-b173-02da9d920934 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:21.544981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daebf61f-b239-4695-b173-02da9d920934 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:21.545417 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daebf61f-b239-4695-b173-02da9d920934 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:21.555014 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:21.621331 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-daebf61f-b239-4695-b173-02da9d920934 None None] MFA Rules not processed for user `d46cd6214aa34547855cec69962f0cfb`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:21.623245 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-daebf61f-b239-4695-b173-02da9d920934 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:21.655723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '42de058800864cb7b2e062ad4bd1fd46', 'request_id': 'req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:21.666346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '42de058800864cb7b2e062ad4bd1fd46', 'request_id': 'req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:21.674981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '42de058800864cb7b2e062ad4bd1fd46', 'request_id': 'req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:21.761671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533 admin admin] The token cache is being invalidate because project 42de058800864cb7b2e062ad4bd1fd46 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:05:21.761671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:21.763653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:21.765065 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 887/1764] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:05:21 2022] DELETE /identity/v3/projects/42de058800864cb7b2e062ad4bd1fd46 => generated 0 bytes in 524 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:21.765990 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:21.772414 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:21.792733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 878/1765] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:21 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 249 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:05:21.794488 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fe4488-8e88-4027-b945-5dd612c89d96 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:21.794904 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fe4488-8e88-4027-b945-5dd612c89d96 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:21.795315 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fe4488-8e88-4027-b945-5dd612c89d96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:21.823711 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-30fe4488-8e88-4027-b945-5dd612c89d96 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:21.825479 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-30fe4488-8e88-4027-b945-5dd612c89d96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:21.917263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 879/1766] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:21.932799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fe4488-8e88-4027-b945-5dd612c89d96 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:21.944294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-30fe4488-8e88-4027-b945-5dd612c89d96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:22.053749 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d13c3c0-3f7b-48b0-9723-cec39d0e3533 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:22.078021 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbc2900-d9a9-40db-8853-92a8842467f3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '123ab6ff796c4e599ce1d6f4095bd898', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:22.078661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbc2900-d9a9-40db-8853-92a8842467f3 service swift] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:22.078974 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbc2900-d9a9-40db-8853-92a8842467f3 service swift] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:22.079218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbc2900-d9a9-40db-8853-92a8842467f3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:22.080171 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fe4488-8e88-4027-b945-5dd612c89d96 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:22.085814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0fbc2900-d9a9-40db-8853-92a8842467f3 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:22.101020 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdc5d2e-8cfe-4841-8b95-93aa4f90d0a0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:22.104157 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdc5d2e-8cfe-4841-8b95-93aa4f90d0a0 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:22.104421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdc5d2e-8cfe-4841-8b95-93aa4f90d0a0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:22.104703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdc5d2e-8cfe-4841-8b95-93aa4f90d0a0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:22.128815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6cdc5d2e-8cfe-4841-8b95-93aa4f90d0a0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:22.179541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdc5d2e-8cfe-4841-8b95-93aa4f90d0a0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:22.181068 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdc5d2e-8cfe-4841-8b95-93aa4f90d0a0 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:22.237363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 880/1767] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:21 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:22.243057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6daa88fb-9b23-4271-bf24-99571541de93 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:22.243057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6daa88fb-9b23-4271-bf24-99571541de93 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:22.243057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6daa88fb-9b23-4271-bf24-99571541de93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:22.318892 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-6daa88fb-9b23-4271-bf24-99571541de93 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:22.320800 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6daa88fb-9b23-4271-bf24-99571541de93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:22.333515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbc2900-d9a9-40db-8853-92a8842467f3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:22.335698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbc2900-d9a9-40db-8853-92a8842467f3 service swift] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:22.343294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 881/1768] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:22.348660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6daa88fb-9b23-4271-bf24-99571541de93 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:22.359007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 888/1769] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 4 01:05:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 593 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 04 01:05:22.372141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6daa88fb-9b23-4271-bf24-99571541de93 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:22.377919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8cdda3-a483-40ca-a15a-1b9c747e9590 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:22.378540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8cdda3-a483-40ca-a15a-1b9c747e9590 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:22.379505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8cdda3-a483-40ca-a15a-1b9c747e9590 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:22.379854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8cdda3-a483-40ca-a15a-1b9c747e9590 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:22.380374 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8cdda3-a483-40ca-a15a-1b9c747e9590 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:05:22.380682 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8cdda3-a483-40ca-a15a-1b9c747e9590 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:22.382351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8cdda3-a483-40ca-a15a-1b9c747e9590 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:22.395685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 882/1770] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:05:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:05:22.433449 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbc2900-d9a9-40db-8853-92a8842467f3 service swift] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:22.468548 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8cdda3-a483-40ca-a15a-1b9c747e9590 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:22.472131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-1e8cdda3-a483-40ca-a15a-1b9c747e9590 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:22.496530 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbc2900-d9a9-40db-8853-92a8842467f3 service swift] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:22.534616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c2bc10-3419-4595-a5de-a7140b5dbf90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:22.545560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c2bc10-3419-4595-a5de-a7140b5dbf90 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:22.545560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c2bc10-3419-4595-a5de-a7140b5dbf90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:22.545560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c2bc10-3419-4595-a5de-a7140b5dbf90 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:22.551989 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c2bc10-3419-4595-a5de-a7140b5dbf90 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:22.568931 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c2bc10-3419-4595-a5de-a7140b5dbf90 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:22.590201 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8cdda3-a483-40ca-a15a-1b9c747e9590 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:22.615824 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:22.629432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:22.629842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:22.630123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5 admin admin] PATH_INFO: `/v3/projects/e4b9f0ce5a8c47cb85cfc09bbeb8450c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:22.661856 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e4b9f0ce5a8c47cb85cfc09bbeb8450c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:22.664052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:22.768498 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 889/1771] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:22 2022] POST /identity/v3/projects => generated 387 bytes in 347 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:22.773344 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c2bc10-3419-4595-a5de-a7140b5dbf90 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:22.921658 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c2bc10-3419-4595-a5de-a7140b5dbf90 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:22.932383 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac64428-fb7a-47d1-92fe-97dbc5e83c2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:22.933454 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac64428-fb7a-47d1-92fe-97dbc5e83c2f admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:22.934047 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac64428-fb7a-47d1-92fe-97dbc5e83c2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:22.936055 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac64428-fb7a-47d1-92fe-97dbc5e83c2f admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:22.937253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac64428-fb7a-47d1-92fe-97dbc5e83c2f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:22.943976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac64428-fb7a-47d1-92fe-97dbc5e83c2f admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:23.023147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e4b9f0ce5a8c47cb85cfc09bbeb8450c', 'request_id': 'req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:23.047209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e4b9f0ce5a8c47cb85cfc09bbeb8450c', 'request_id': 'req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:23.224843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 890/1772] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:22 2022] POST /identity/v3/users => generated 294 bytes in 452 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:05:23.228926 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac64428-fb7a-47d1-92fe-97dbc5e83c2f admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:23.244464 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac64428-fb7a-47d1-92fe-97dbc5e83c2f admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:23.259970 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e4b9f0ce5a8c47cb85cfc09bbeb8450c', 'request_id': 'req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:23.297035 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5 admin admin] The token cache is being invalidate because project e4b9f0ce5a8c47cb85cfc09bbeb8450c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:05:23.297699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:23.301452 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:23.309889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 883/1773] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:05:22 2022] DELETE /identity/v3/projects/e4b9f0ce5a8c47cb85cfc09bbeb8450c => generated 0 bytes in 844 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:23.311437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:23.313842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:23.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:23.493952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0bd32e-e2d4-473f-a2c9-6ad6fa28caf5 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:23.505256 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764e1484-0944-4a95-b1e5-a61b8d51366a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:23.506455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764e1484-0944-4a95-b1e5-a61b8d51366a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:23.509347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764e1484-0944-4a95-b1e5-a61b8d51366a admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:23.509972 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764e1484-0944-4a95-b1e5-a61b8d51366a admin admin] PATH_INFO: `/v3/users/8d501ad64e734d8b9e5378a17eef9f42` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:23.515603 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764e1484-0944-4a95-b1e5-a61b8d51366a admin admin] RBAC: Authorizing `identity:delete_user(user_id=8d501ad64e734d8b9e5378a17eef9f42)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:23.526860 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764e1484-0944-4a95-b1e5-a61b8d51366a admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:23.533892 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206f8f8c-b171-4326-a079-c68f10fb17b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:23.534540 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206f8f8c-b171-4326-a079-c68f10fb17b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:23.535360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206f8f8c-b171-4326-a079-c68f10fb17b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:23.535643 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206f8f8c-b171-4326-a079-c68f10fb17b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:23.536307 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206f8f8c-b171-4326-a079-c68f10fb17b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:23.542640 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206f8f8c-b171-4326-a079-c68f10fb17b4 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:23.582590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 891/1774] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:23 2022] GET /identity/v3/roles => generated 1420 bytes in 354 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:05:23.586749 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206f8f8c-b171-4326-a079-c68f10fb17b4 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:23.589984 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-206f8f8c-b171-4326-a079-c68f10fb17b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:23.632409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:23.637299 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206f8f8c-b171-4326-a079-c68f10fb17b4 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:23.646494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05f2b3d-d1f2-4e35-b566-ae887716c165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:23.647040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05f2b3d-d1f2-4e35-b566-ae887716c165 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:23.647407 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05f2b3d-d1f2-4e35-b566-ae887716c165 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:23.647830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05f2b3d-d1f2-4e35-b566-ae887716c165 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:23.648621 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05f2b3d-d1f2-4e35-b566-ae887716c165 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:23.650520 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05f2b3d-d1f2-4e35-b566-ae887716c165 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:23.670469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 892/1775] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:23 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:23.674847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05f2b3d-d1f2-4e35-b566-ae887716c165 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:23.737499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05f2b3d-d1f2-4e35-b566-ae887716c165 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:23.756154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ba0704-3c82-4b8c-a9bd-0601e67c04d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:23.757001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ba0704-3c82-4b8c-a9bd-0601e67c04d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:23.768995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ba0704-3c82-4b8c-a9bd-0601e67c04d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:23.768995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ba0704-3c82-4b8c-a9bd-0601e67c04d0 admin admin] PATH_INFO: `/v3/projects/3ea5113c49d4461695f17b1499fc7b43/users/e896bc22947048b484dbc425f5f49664/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:23.789754 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ba0704-3c82-4b8c-a9bd-0601e67c04d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3ea5113c49d4461695f17b1499fc7b43, user_id=e896bc22947048b484dbc425f5f49664, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:23.805209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ba0704-3c82-4b8c-a9bd-0601e67c04d0 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:23.873645 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-764e1484-0944-4a95-b1e5-a61b8d51366a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8d501ad64e734d8b9e5378a17eef9f42'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:23.874910 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-764e1484-0944-4a95-b1e5-a61b8d51366a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8d501ad64e734d8b9e5378a17eef9f42'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:23.952112 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-764e1484-0944-4a95-b1e5-a61b8d51366a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8d501ad64e734d8b9e5378a17eef9f42'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:23.965791 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 893/1776] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:23 2022] PUT /identity/v3/projects/3ea5113c49d4461695f17b1499fc7b43/users/e896bc22947048b484dbc425f5f49664/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:05:23.976655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75af9622-83b1-4d63-b181-20bd6fb6ec2a None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:23.977491 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75af9622-83b1-4d63-b181-20bd6fb6ec2a None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:23.977978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75af9622-83b1-4d63-b181-20bd6fb6ec2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:23.995814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-764e1484-0944-4a95-b1e5-a61b8d51366a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8d501ad64e734d8b9e5378a17eef9f42'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:23.997883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-764e1484-0944-4a95-b1e5-a61b8d51366a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8d501ad64e734d8b9e5378a17eef9f42'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:24.082930 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 884/1777] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:05:23 2022] DELETE /identity/v3/users/8d501ad64e734d8b9e5378a17eef9f42 => generated 0 bytes in 772 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:24.108469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-75af9622-83b1-4d63-b181-20bd6fb6ec2a None None] MFA Rules not processed for user `e896bc22947048b484dbc425f5f49664`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:24.110817 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-75af9622-83b1-4d63-b181-20bd6fb6ec2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:24.267662 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 894/1778] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:23 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 292 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:05:24.272436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764e1484-0944-4a95-b1e5-a61b8d51366a admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:24.274232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-764e1484-0944-4a95-b1e5-a61b8d51366a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:24.378859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764e1484-0944-4a95-b1e5-a61b8d51366a admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:24.384336 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f1246d-5325-4790-9b98-4a850e1f2ed5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:24.385107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f1246d-5325-4790-9b98-4a850e1f2ed5 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:24.388401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f1246d-5325-4790-9b98-4a850e1f2ed5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:24.388401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f1246d-5325-4790-9b98-4a850e1f2ed5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:24.388401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f1246d-5325-4790-9b98-4a850e1f2ed5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:24.389246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f1246d-5325-4790-9b98-4a850e1f2ed5 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:24.580723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 885/1779] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:24 2022] POST /identity/v3/projects => generated 389 bytes in 309 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:05:24.587112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75af9622-83b1-4d63-b181-20bd6fb6ec2a None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:24.629781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6c2d40-632f-4a00-8b61-e975941f4d54 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:24.629781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6c2d40-632f-4a00-8b61-e975941f4d54 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:24.632196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6c2d40-632f-4a00-8b61-e975941f4d54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:24.652843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75af9622-83b1-4d63-b181-20bd6fb6ec2a None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:24.733943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-0c6c2d40-632f-4a00-8b61-e975941f4d54 None None] MFA Rules not processed for user `44993cd93360442bb0afb28778ee83bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:24.746240 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-0c6c2d40-632f-4a00-8b61-e975941f4d54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:24.916941 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d771a1-e46b-40ba-8ce6-a8de933cac8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:24.940705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d771a1-e46b-40ba-8ce6-a8de933cac8e admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:24.945094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d771a1-e46b-40ba-8ce6-a8de933cac8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:24.945094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d771a1-e46b-40ba-8ce6-a8de933cac8e admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:24.945094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d771a1-e46b-40ba-8ce6-a8de933cac8e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:24.958704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d771a1-e46b-40ba-8ce6-a8de933cac8e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:24.972938 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 886/1780] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:24 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 345 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:24.988798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:25.170800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 895/1781] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:24 2022] POST /identity/v3/users => generated 301 bytes in 584 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:25.175705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6c2d40-632f-4a00-8b61-e975941f4d54 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:25.198216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6c2d40-632f-4a00-8b61-e975941f4d54 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:25.224934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cbe4ee-72b2-4458-86db-498b21ce91b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:25.225607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cbe4ee-72b2-4458-86db-498b21ce91b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:25.225889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cbe4ee-72b2-4458-86db-498b21ce91b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:25.226163 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cbe4ee-72b2-4458-86db-498b21ce91b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:25.227072 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cbe4ee-72b2-4458-86db-498b21ce91b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:25.229226 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cbe4ee-72b2-4458-86db-498b21ce91b0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:25.260449 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d771a1-e46b-40ba-8ce6-a8de933cac8e admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:25.286260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 887/1782] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:25 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:25.291568 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d771a1-e46b-40ba-8ce6-a8de933cac8e admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:25.305538 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5f1061-54c7-4508-aaa9-287d0d77737d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:25.306469 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5f1061-54c7-4508-aaa9-287d0d77737d admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:25.309575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5f1061-54c7-4508-aaa9-287d0d77737d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:25.310069 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5f1061-54c7-4508-aaa9-287d0d77737d admin admin] PATH_INFO: `/v3/projects/42233f43b574453eb16ec7c6c7437905/users/6ca095fb437447c4bdac0599d3783b17/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:25.350287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5f1061-54c7-4508-aaa9-287d0d77737d admin admin] RBAC: Authorizing `identity:create_grant(project_id=42233f43b574453eb16ec7c6c7437905, user_id=6ca095fb437447c4bdac0599d3783b17, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:25.353756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5f1061-54c7-4508-aaa9-287d0d77737d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:25.393606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cbe4ee-72b2-4458-86db-498b21ce91b0 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:25.396763 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-65cbe4ee-72b2-4458-86db-498b21ce91b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:25.445131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cbe4ee-72b2-4458-86db-498b21ce91b0 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:25.497981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 896/1783] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:25 2022] PUT /identity/v3/projects/42233f43b574453eb16ec7c6c7437905/users/6ca095fb437447c4bdac0599d3783b17/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:25.504356 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5f1061-54c7-4508-aaa9-287d0d77737d admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:25.518960 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6070ac50-c9c5-4700-9502-af45c5c2956d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:25.521645 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6070ac50-c9c5-4700-9502-af45c5c2956d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:25.524531 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6070ac50-c9c5-4700-9502-af45c5c2956d admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:25.525298 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6070ac50-c9c5-4700-9502-af45c5c2956d admin admin] PATH_INFO: `/v3/projects/59e345a392104034b113adab2fa35c8f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:25.531172 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6070ac50-c9c5-4700-9502-af45c5c2956d admin admin] RBAC: Authorizing `identity:delete_project(project_id=59e345a392104034b113adab2fa35c8f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:25.544213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6070ac50-c9c5-4700-9502-af45c5c2956d admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:25.826628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5f1061-54c7-4508-aaa9-287d0d77737d admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:25.849311 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-6070ac50-c9c5-4700-9502-af45c5c2956d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '59e345a392104034b113adab2fa35c8f', 'request_id': 'req-6070ac50-c9c5-4700-9502-af45c5c2956d'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:25.883118 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-6070ac50-c9c5-4700-9502-af45c5c2956d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '59e345a392104034b113adab2fa35c8f', 'request_id': 'req-6070ac50-c9c5-4700-9502-af45c5c2956d'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:25.982397 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-6070ac50-c9c5-4700-9502-af45c5c2956d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '59e345a392104034b113adab2fa35c8f', 'request_id': 'req-6070ac50-c9c5-4700-9502-af45c5c2956d'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:25.997723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9f9cad-4dcb-4d76-adf8-bbdf8663708f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:25.999382 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9f9cad-4dcb-4d76-adf8-bbdf8663708f admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:26.000179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9f9cad-4dcb-4d76-adf8-bbdf8663708f admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:26.000752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9f9cad-4dcb-4d76-adf8-bbdf8663708f admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:26.002278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9f9cad-4dcb-4d76-adf8-bbdf8663708f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:26.008702 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9f9cad-4dcb-4d76-adf8-bbdf8663708f admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:26.016447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-6070ac50-c9c5-4700-9502-af45c5c2956d admin admin] The token cache is being invalidate because project 59e345a392104034b113adab2fa35c8f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:05:26.018039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-6070ac50-c9c5-4700-9502-af45c5c2956d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:26.020272 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-6070ac50-c9c5-4700-9502-af45c5c2956d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:26.023815 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 897/1785] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:25 2022] GET /identity/v3/roles => generated 1420 bytes in 521 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:26.027270 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:26.028907 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5a6ed4-c68e-4a54-9700-cdc40c2a72af None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:26.029315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5a6ed4-c68e-4a54-9700-cdc40c2a72af None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:26.030125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5a6ed4-c68e-4a54-9700-cdc40c2a72af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:26.030702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 888/1785] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:05:25 2022] DELETE /identity/v3/projects/59e345a392104034b113adab2fa35c8f => generated 0 bytes in 633 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:26.031836 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6070ac50-c9c5-4700-9502-af45c5c2956d admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:26.034023 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6070ac50-c9c5-4700-9502-af45c5c2956d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:26.060800 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6070ac50-c9c5-4700-9502-af45c5c2956d admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:26.070218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502e88ab-dd9a-4f28-a0b4-dc088c3a6fbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:26.073283 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-ce5a6ed4-c68e-4a54-9700-cdc40c2a72af None None] MFA Rules not processed for user `da4f44b86ca94adf959d9a41d915c902`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:26.075691 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ce5a6ed4-c68e-4a54-9700-cdc40c2a72af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:26.108843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502e88ab-dd9a-4f28-a0b4-dc088c3a6fbe admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:26.109290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502e88ab-dd9a-4f28-a0b4-dc088c3a6fbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:26.113457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502e88ab-dd9a-4f28-a0b4-dc088c3a6fbe admin admin] PATH_INFO: `/v3/projects/42233f43b574453eb16ec7c6c7437905/users/6ca095fb437447c4bdac0599d3783b17/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:26.113457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502e88ab-dd9a-4f28-a0b4-dc088c3a6fbe admin admin] RBAC: Authorizing `identity:create_grant(project_id=42233f43b574453eb16ec7c6c7437905, user_id=6ca095fb437447c4bdac0599d3783b17, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:26.115605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-502e88ab-dd9a-4f28-a0b4-dc088c3a6fbe admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:26.139288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 889/1786] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:26 2022] PUT /identity/v3/projects/42233f43b574453eb16ec7c6c7437905/users/6ca095fb437447c4bdac0599d3783b17/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:26.141250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499ef865-be5d-4839-8fcc-cfa30573ca6e None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:26.141250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499ef865-be5d-4839-8fcc-cfa30573ca6e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:26.141250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499ef865-be5d-4839-8fcc-cfa30573ca6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:26.172027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-499ef865-be5d-4839-8fcc-cfa30573ca6e None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:26.179722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-499ef865-be5d-4839-8fcc-cfa30573ca6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:26.199154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 898/1787] 10.222.0.16 () {58 vars in 1015 bytes} [Sun Sep 4 01:05:26 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:26.203219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e289a6-ae26-4501-a6af-0f9eea48148b None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:26.203992 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e289a6-ae26-4501-a6af-0f9eea48148b None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:26.204459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e289a6-ae26-4501-a6af-0f9eea48148b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:26.350124 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-10e289a6-ae26-4501-a6af-0f9eea48148b None None] MFA Rules not processed for user `6ca095fb437447c4bdac0599d3783b17`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:26.350124 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-10e289a6-ae26-4501-a6af-0f9eea48148b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:26.393052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 890/1788] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:26.421633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bd9d8c-1a6b-4388-8821-9b32dd924ae1 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:26.421633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bd9d8c-1a6b-4388-8821-9b32dd924ae1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:26.423900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bd9d8c-1a6b-4388-8821-9b32dd924ae1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:26.461955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-77bd9d8c-1a6b-4388-8821-9b32dd924ae1 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:26.464029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-77bd9d8c-1a6b-4388-8821-9b32dd924ae1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:26.481348 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 891/1789] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:26.482594 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bd9d8c-1a6b-4388-8821-9b32dd924ae1 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:26.486006 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-77bd9d8c-1a6b-4388-8821-9b32dd924ae1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:26.630325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bd9d8c-1a6b-4388-8821-9b32dd924ae1 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:26.635145 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c46b849-a52b-44c3-a3a9-11745d96b3b2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:26.635823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c46b849-a52b-44c3-a3a9-11745d96b3b2 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:26.636309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c46b849-a52b-44c3-a3a9-11745d96b3b2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:26.636596 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c46b849-a52b-44c3-a3a9-11745d96b3b2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:26.652624 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c46b849-a52b-44c3-a3a9-11745d96b3b2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:26.654169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c46b849-a52b-44c3-a3a9-11745d96b3b2 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:26.677043 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 892/1790] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:26 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:26.678748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c46b849-a52b-44c3-a3a9-11745d96b3b2 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:26.691833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c46b849-a52b-44c3-a3a9-11745d96b3b2 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:26.742827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03756da1-e2a6-48b8-8380-92b22fd04177 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:26.742827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03756da1-e2a6-48b8-8380-92b22fd04177 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:26.742827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03756da1-e2a6-48b8-8380-92b22fd04177 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:26.742827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03756da1-e2a6-48b8-8380-92b22fd04177 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:26.743470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03756da1-e2a6-48b8-8380-92b22fd04177 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:05:26.743470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03756da1-e2a6-48b8-8380-92b22fd04177 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:26.743470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03756da1-e2a6-48b8-8380-92b22fd04177 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:26.743711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 899/1791] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:26 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 533 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:26.743815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 893/1792] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:05:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:05:26.766020 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e289a6-ae26-4501-a6af-0f9eea48148b None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:26.771908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03756da1-e2a6-48b8-8380-92b22fd04177 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:26.774051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-03756da1-e2a6-48b8-8380-92b22fd04177 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:26.809597 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e289a6-ae26-4501-a6af-0f9eea48148b None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:26.844892 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38c6cb3-8665-4618-9923-2b62dfe3f095 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:26.844892 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38c6cb3-8665-4618-9923-2b62dfe3f095 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:26.844892 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38c6cb3-8665-4618-9923-2b62dfe3f095 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:26.844892 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38c6cb3-8665-4618-9923-2b62dfe3f095 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:26.845519 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38c6cb3-8665-4618-9923-2b62dfe3f095 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:26.845519 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38c6cb3-8665-4618-9923-2b62dfe3f095 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:26.917629 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03756da1-e2a6-48b8-8380-92b22fd04177 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:26.926559 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7695575e-b162-458c-9166-588f28033211 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:26.930530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7695575e-b162-458c-9166-588f28033211 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:26.931254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7695575e-b162-458c-9166-588f28033211 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:26.931683 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7695575e-b162-458c-9166-588f28033211 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:26.963779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7695575e-b162-458c-9166-588f28033211 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:26.969650 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7695575e-b162-458c-9166-588f28033211 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:26.996389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 900/1793] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:26 2022] POST /identity/v3/projects => generated 377 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:27.002059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38c6cb3-8665-4618-9923-2b62dfe3f095 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:27.062224 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 894/1794] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:26 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 286 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:27.231493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38c6cb3-8665-4618-9923-2b62dfe3f095 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:27.244805 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a24805a-457f-41df-b4d5-d922920e1344 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:27.249250 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a24805a-457f-41df-b4d5-d922920e1344 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:27.249725 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a24805a-457f-41df-b4d5-d922920e1344 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:27.253989 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a24805a-457f-41df-b4d5-d922920e1344 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:27.253989 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a24805a-457f-41df-b4d5-d922920e1344 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:27.253989 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a24805a-457f-41df-b4d5-d922920e1344 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:27.374321 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 901/1795] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:26 2022] POST /identity/v3/users => generated 289 bytes in 371 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:27.380140 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7695575e-b162-458c-9166-588f28033211 service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:27.391964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7695575e-b162-458c-9166-588f28033211 service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:27.414137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159b99da-d9d6-4344-91df-bf0b73297c3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:27.414946 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159b99da-d9d6-4344-91df-bf0b73297c3b admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:27.415225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159b99da-d9d6-4344-91df-bf0b73297c3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:27.415497 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159b99da-d9d6-4344-91df-bf0b73297c3b admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:27.416153 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159b99da-d9d6-4344-91df-bf0b73297c3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:27.417830 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159b99da-d9d6-4344-91df-bf0b73297c3b admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:27.481823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 895/1796] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:27 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:27.485969 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a24805a-457f-41df-b4d5-d922920e1344 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:27.502421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a24805a-457f-41df-b4d5-d922920e1344 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:27.540434 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f4a9a2-5d35-435a-97f3-88ab4a6046ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:27.544313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f4a9a2-5d35-435a-97f3-88ab4a6046ab admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:27.545626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f4a9a2-5d35-435a-97f3-88ab4a6046ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:27.546828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f4a9a2-5d35-435a-97f3-88ab4a6046ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:27.547710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f4a9a2-5d35-435a-97f3-88ab4a6046ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:27.556481 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f4a9a2-5d35-435a-97f3-88ab4a6046ab admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:27.571168 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159b99da-d9d6-4344-91df-bf0b73297c3b admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:27.585870 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159b99da-d9d6-4344-91df-bf0b73297c3b admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:27.595494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:27.595494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 902/1797] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:27 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:27.598989 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f4a9a2-5d35-435a-97f3-88ab4a6046ab admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:27.599627 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4731e33-a651-4cda-8c3d-3272f1855fa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:27.600392 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4731e33-a651-4cda-8c3d-3272f1855fa0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:27.600674 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4731e33-a651-4cda-8c3d-3272f1855fa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:27.601018 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4731e33-a651-4cda-8c3d-3272f1855fa0 admin admin] PATH_INFO: `/v3/users/e896bc22947048b484dbc425f5f49664` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:27.602580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4731e33-a651-4cda-8c3d-3272f1855fa0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e896bc22947048b484dbc425f5f49664)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:27.604499 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4731e33-a651-4cda-8c3d-3272f1855fa0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:27.611439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f4a9a2-5d35-435a-97f3-88ab4a6046ab admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:27.623897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d51a3f-aa2a-4e22-a28e-a637cb42ecc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:27.627304 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d51a3f-aa2a-4e22-a28e-a637cb42ecc8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:27.627627 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d51a3f-aa2a-4e22-a28e-a637cb42ecc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:27.629518 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d51a3f-aa2a-4e22-a28e-a637cb42ecc8 admin admin] PATH_INFO: `/v3/projects/5d8ec7c6ee57418eb3a95d2e0b857f75/users/5d1a4360c16d4b49a59baf59622b6d16/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:27.655549 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:27.684394 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d51a3f-aa2a-4e22-a28e-a637cb42ecc8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d8ec7c6ee57418eb3a95d2e0b857f75, user_id=5d1a4360c16d4b49a59baf59622b6d16, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:27.687437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d51a3f-aa2a-4e22-a28e-a637cb42ecc8 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:27.789658 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 903/1798] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:27 2022] PUT /identity/v3/projects/5d8ec7c6ee57418eb3a95d2e0b857f75/users/5d1a4360c16d4b49a59baf59622b6d16/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:27.801028 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fa5a7e-c4f8-499e-be5b-5152ec24cbcd None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:27.802036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fa5a7e-c4f8-499e-be5b-5152ec24cbcd None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:27.805818 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fa5a7e-c4f8-499e-be5b-5152ec24cbcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:27.927452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-49fa5a7e-c4f8-499e-be5b-5152ec24cbcd None None] MFA Rules not processed for user `5d1a4360c16d4b49a59baf59622b6d16`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:27.929108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b4731e33-a651-4cda-8c3d-3272f1855fa0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e896bc22947048b484dbc425f5f49664'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:27.932826 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-49fa5a7e-c4f8-499e-be5b-5152ec24cbcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:27.933471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b4731e33-a651-4cda-8c3d-3272f1855fa0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e896bc22947048b484dbc425f5f49664'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:28.010958 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b4731e33-a651-4cda-8c3d-3272f1855fa0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e896bc22947048b484dbc425f5f49664'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:28.054411 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b4731e33-a651-4cda-8c3d-3272f1855fa0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e896bc22947048b484dbc425f5f49664'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:28.056564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b4731e33-a651-4cda-8c3d-3272f1855fa0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e896bc22947048b484dbc425f5f49664'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:28.121799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 896/1799] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:05:27 2022] DELETE /identity/v3/users/e896bc22947048b484dbc425f5f49664 => generated 0 bytes in 551 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:28.127782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be94fd3-c385-4947-b6e8-65e976bb4b76 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:28.127782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be94fd3-c385-4947-b6e8-65e976bb4b76 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:28.128080 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be94fd3-c385-4947-b6e8-65e976bb4b76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:28.172834 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 904/1800] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:27 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 373 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:28.176959 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe81ff66-e9ed-4573-af38-95c19163f461 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:28.177441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe81ff66-e9ed-4573-af38-95c19163f461 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:28.177869 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe81ff66-e9ed-4573-af38-95c19163f461 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:28.219933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-6be94fd3-c385-4947-b6e8-65e976bb4b76 None None] MFA Rules not processed for user `f5e4723258c64ce5885154dded2d3d43`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:28.221235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6be94fd3-c385-4947-b6e8-65e976bb4b76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:28.256582 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-fe81ff66-e9ed-4573-af38-95c19163f461 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:28.260279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-fe81ff66-e9ed-4573-af38-95c19163f461 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:28.434119 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 897/1801] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:28 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 284 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:28.434119 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be94fd3-c385-4947-b6e8-65e976bb4b76 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:28.434119 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6be94fd3-c385-4947-b6e8-65e976bb4b76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:28.441701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 905/1802] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:28.448085 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be94fd3-c385-4947-b6e8-65e976bb4b76 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:28.451453 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe81ff66-e9ed-4573-af38-95c19163f461 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:28.463117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-fe81ff66-e9ed-4573-af38-95c19163f461 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:28.468013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254b980e-9618-4deb-9046-5b6b940af0da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:28.469280 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254b980e-9618-4deb-9046-5b6b940af0da admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:28.470095 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254b980e-9618-4deb-9046-5b6b940af0da admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:28.470584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254b980e-9618-4deb-9046-5b6b940af0da admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:28.471477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254b980e-9618-4deb-9046-5b6b940af0da admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:28.474350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254b980e-9618-4deb-9046-5b6b940af0da admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:28.516527 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe81ff66-e9ed-4573-af38-95c19163f461 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:28.540916 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a935e4-2b01-4165-a17a-22cace46943a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:28.541646 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a935e4-2b01-4165-a17a-22cace46943a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:28.541991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a935e4-2b01-4165-a17a-22cace46943a admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:28.542325 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a935e4-2b01-4165-a17a-22cace46943a admin admin] PATH_INFO: `/v3/users/8bf3e8f154574a648bfb56e6882d617a` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:28.551561 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a935e4-2b01-4165-a17a-22cace46943a admin admin] RBAC: Authorizing `identity:delete_user(user_id=8bf3e8f154574a648bfb56e6882d617a)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:28.553698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a935e4-2b01-4165-a17a-22cace46943a admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:28.619385 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 898/1803] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:28 2022] POST /identity/v3/projects => generated 377 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:28.620452 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254b980e-9618-4deb-9046-5b6b940af0da admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:28.626665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-254b980e-9618-4deb-9046-5b6b940af0da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:28.824305 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254b980e-9618-4deb-9046-5b6b940af0da admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:28.829147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998bf8f6-8bdf-466d-861a-091ad1b67853 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:28.830017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998bf8f6-8bdf-466d-861a-091ad1b67853 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:28.830454 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998bf8f6-8bdf-466d-861a-091ad1b67853 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:28.830896 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998bf8f6-8bdf-466d-861a-091ad1b67853 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:29.001287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-90a935e4-2b01-4165-a17a-22cace46943a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8bf3e8f154574a648bfb56e6882d617a'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:29.024502 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998bf8f6-8bdf-466d-861a-091ad1b67853 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:29.030046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998bf8f6-8bdf-466d-861a-091ad1b67853 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:29.064150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 899/1804] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:29.065809 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998bf8f6-8bdf-466d-861a-091ad1b67853 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:29.095548 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998bf8f6-8bdf-466d-861a-091ad1b67853 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:29.123903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156f9814-5ff6-425c-9d1c-f238493450d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:29.124602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156f9814-5ff6-425c-9d1c-f238493450d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:29.124891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156f9814-5ff6-425c-9d1c-f238493450d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:29.125204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156f9814-5ff6-425c-9d1c-f238493450d5 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:29.133175 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156f9814-5ff6-425c-9d1c-f238493450d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:29.135160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156f9814-5ff6-425c-9d1c-f238493450d5 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:29.139258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-90a935e4-2b01-4165-a17a-22cace46943a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8bf3e8f154574a648bfb56e6882d617a'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:29.167667 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-90a935e4-2b01-4165-a17a-22cace46943a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8bf3e8f154574a648bfb56e6882d617a'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:29.168420 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-90a935e4-2b01-4165-a17a-22cace46943a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8bf3e8f154574a648bfb56e6882d617a'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:29.184343 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-90a935e4-2b01-4165-a17a-22cace46943a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8bf3e8f154574a648bfb56e6882d617a'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:29.214345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 906/1805] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:05:28 2022] DELETE /identity/v3/users/8bf3e8f154574a648bfb56e6882d617a => generated 0 bytes in 769 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:29.215998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec18af87-e1a6-48c6-b0cf-790e5de9928d None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:29.216301 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec18af87-e1a6-48c6-b0cf-790e5de9928d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:29.216581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec18af87-e1a6-48c6-b0cf-790e5de9928d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:29.269832 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-ec18af87-e1a6-48c6-b0cf-790e5de9928d None None] MFA Rules not processed for user `fbcd4c19e766446188e3b3f41a44e2e4`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:29.278637 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ec18af87-e1a6-48c6-b0cf-790e5de9928d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:29.345240 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 900/1806] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:29 2022] POST /identity/v3/users => generated 295 bytes in 278 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:29.351339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457fd23-e854-43bb-abc3-9d4fae81eef7 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:29.351639 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457fd23-e854-43bb-abc3-9d4fae81eef7 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:29.351923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457fd23-e854-43bb-abc3-9d4fae81eef7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:29.425101 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-3457fd23-e854-43bb-abc3-9d4fae81eef7 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:29.433484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-3457fd23-e854-43bb-abc3-9d4fae81eef7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:29.539861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 907/1807] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:29 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 325 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:29.544146 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec18af87-e1a6-48c6-b0cf-790e5de9928d None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:29.549831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ec18af87-e1a6-48c6-b0cf-790e5de9928d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:29.572564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 901/1808] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:29.622776 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec18af87-e1a6-48c6-b0cf-790e5de9928d None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:29.636541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebc3143-383a-41a5-913d-180c1a6a2ba0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:29.639024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebc3143-383a-41a5-913d-180c1a6a2ba0 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:29.639434 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebc3143-383a-41a5-913d-180c1a6a2ba0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:29.641249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebc3143-383a-41a5-913d-180c1a6a2ba0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:29.642108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3457fd23-e854-43bb-abc3-9d4fae81eef7 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:29.643697 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-3457fd23-e854-43bb-abc3-9d4fae81eef7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:29.644755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebc3143-383a-41a5-913d-180c1a6a2ba0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:29.649976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebc3143-383a-41a5-913d-180c1a6a2ba0 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:29.684733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 908/1809] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:29 2022] GET /identity/v3/roles => generated 1420 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:29.770792 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebc3143-383a-41a5-913d-180c1a6a2ba0 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:29.780562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3457fd23-e854-43bb-abc3-9d4fae81eef7 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:29.784739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c135f04e-031d-421e-b0a8-b5041094bd03 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:29.785014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebc3143-383a-41a5-913d-180c1a6a2ba0 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:29.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c135f04e-031d-421e-b0a8-b5041094bd03 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:29.786157 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c135f04e-031d-421e-b0a8-b5041094bd03 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:29.786509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c135f04e-031d-421e-b0a8-b5041094bd03 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:29.798114 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fba9c6b-7d61-4964-aa33-6bc0c05da9c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:29.798903 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fba9c6b-7d61-4964-aa33-6bc0c05da9c8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:29.804419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fba9c6b-7d61-4964-aa33-6bc0c05da9c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:29.804900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fba9c6b-7d61-4964-aa33-6bc0c05da9c8 admin admin] PATH_INFO: `/v3/projects/d30baef84b62499fac7cfe20946356ec/users/8e0207d52df145e0b44afe10d59f8ad6/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:29.808057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c135f04e-031d-421e-b0a8-b5041094bd03 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:29.816462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c135f04e-031d-421e-b0a8-b5041094bd03 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:29.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 902/1810] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:29.840692 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fba9c6b-7d61-4964-aa33-6bc0c05da9c8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d30baef84b62499fac7cfe20946356ec, user_id=8e0207d52df145e0b44afe10d59f8ad6, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:29.848835 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fba9c6b-7d61-4964-aa33-6bc0c05da9c8 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:29.956506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 909/1811] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:29 2022] PUT /identity/v3/projects/d30baef84b62499fac7cfe20946356ec/users/8e0207d52df145e0b44afe10d59f8ad6/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:29.957466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c135f04e-031d-421e-b0a8-b5041094bd03 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:29.979322 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c135f04e-031d-421e-b0a8-b5041094bd03 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:30.195567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c6cd96-4f8c-46ed-bb5a-b10bf7660892 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:30.197473 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c6cd96-4f8c-46ed-bb5a-b10bf7660892 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:30.198440 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c6cd96-4f8c-46ed-bb5a-b10bf7660892 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:30.198440 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c6cd96-4f8c-46ed-bb5a-b10bf7660892 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:30.198911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c6cd96-4f8c-46ed-bb5a-b10bf7660892 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:30.201238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c6cd96-4f8c-46ed-bb5a-b10bf7660892 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:30.221340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fba9c6b-7d61-4964-aa33-6bc0c05da9c8 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:30.233748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 903/1812] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:29 2022] GET /identity/v3/roles => generated 1420 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:30.238812 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fba9c6b-7d61-4964-aa33-6bc0c05da9c8 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:30.246231 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4ce453-3710-4cf0-9a0f-e0bcce0ed518 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:30.247324 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4ce453-3710-4cf0-9a0f-e0bcce0ed518 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:30.247778 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4ce453-3710-4cf0-9a0f-e0bcce0ed518 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:30.248132 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4ce453-3710-4cf0-9a0f-e0bcce0ed518 admin admin] PATH_INFO: `/v3/projects/d30baef84b62499fac7cfe20946356ec/users/8e0207d52df145e0b44afe10d59f8ad6/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:30.250583 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4ce453-3710-4cf0-9a0f-e0bcce0ed518 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d30baef84b62499fac7cfe20946356ec, user_id=8e0207d52df145e0b44afe10d59f8ad6, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:30.254481 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4ce453-3710-4cf0-9a0f-e0bcce0ed518 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:30.270512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 910/1813] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:30 2022] PUT /identity/v3/projects/d30baef84b62499fac7cfe20946356ec/users/8e0207d52df145e0b44afe10d59f8ad6/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:05:30.282741 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1935037-26dd-4535-9734-fe115d4f0c9a None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:30.283208 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1935037-26dd-4535-9734-fe115d4f0c9a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:30.283622 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1935037-26dd-4535-9734-fe115d4f0c9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:30.336659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-b1935037-26dd-4535-9734-fe115d4f0c9a None None] MFA Rules not processed for user `8e0207d52df145e0b44afe10d59f8ad6`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:30.339246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b1935037-26dd-4535-9734-fe115d4f0c9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:30.458555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 904/1814] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:30 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:30.465951 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4ce453-3710-4cf0-9a0f-e0bcce0ed518 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:30.469907 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-5f4ce453-3710-4cf0-9a0f-e0bcce0ed518 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:30.593748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4ce453-3710-4cf0-9a0f-e0bcce0ed518 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:30.640346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b89348c-e4b5-4881-8bc9-88d44594bdcd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:30.640346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b89348c-e4b5-4881-8bc9-88d44594bdcd service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:30.640346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b89348c-e4b5-4881-8bc9-88d44594bdcd service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:30.640346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b89348c-e4b5-4881-8bc9-88d44594bdcd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:30.640801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9b89348c-e4b5-4881-8bc9-88d44594bdcd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:30.709491 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b89348c-e4b5-4881-8bc9-88d44594bdcd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:30.713042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b89348c-e4b5-4881-8bc9-88d44594bdcd service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:30.724178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 911/1815] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:30 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:30.731034 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:DELETE=4,keystone:SELECT=94,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:30.822452 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:31.044322 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ed59b7-7a1d-4b21-af0f-1cd4872aa98f None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:31.045807 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ed59b7-7a1d-4b21-af0f-1cd4872aa98f None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:31.046147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ed59b7-7a1d-4b21-af0f-1cd4872aa98f None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:31.048720 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 905/1816] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 4 01:05:31 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:05:31.193529 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c63ca4-94ef-4cc3-92c8-b615a8cef7de None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:31.193903 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c63ca4-94ef-4cc3-92c8-b615a8cef7de None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:31.194287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c63ca4-94ef-4cc3-92c8-b615a8cef7de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:31.260310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-d8c63ca4-94ef-4cc3-92c8-b615a8cef7de None None] MFA Rules not processed for user `57c88f16c7df40adb917f49804fda21f`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:31.264357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d8c63ca4-94ef-4cc3-92c8-b615a8cef7de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:31.388227 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 912/1817] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 4 01:05:31 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:32.758894 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ed59b7-7a1d-4b21-af0f-1cd4872aa98f None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:32.758894 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d9ed59b7-7a1d-4b21-af0f-1cd4872aa98f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:32.789016 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ed59b7-7a1d-4b21-af0f-1cd4872aa98f None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:32.801856 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae94ba3-6081-4737-843f-1c20b9c348cc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:32.801856 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae94ba3-6081-4737-843f-1c20b9c348cc service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:32.801856 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae94ba3-6081-4737-843f-1c20b9c348cc service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:32.801856 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae94ba3-6081-4737-843f-1c20b9c348cc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:32.802873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c63ca4-94ef-4cc3-92c8-b615a8cef7de None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:32.804428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d8c63ca4-94ef-4cc3-92c8-b615a8cef7de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:32.833980 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae94ba3-6081-4737-843f-1c20b9c348cc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:32.835591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae94ba3-6081-4737-843f-1c20b9c348cc service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:32.849107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 906/1818] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:33.063038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c63ca4-94ef-4cc3-92c8-b615a8cef7de None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:33.063038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bf1013-1bfe-4445-9d46-f5625067652e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:33.063038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bf1013-1bfe-4445-9d46-f5625067652e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:33.063038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bf1013-1bfe-4445-9d46-f5625067652e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:33.064193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bf1013-1bfe-4445-9d46-f5625067652e admin admin] PATH_INFO: `/v3/projects/3ea5113c49d4461695f17b1499fc7b43` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:33.064193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1bf1013-1bfe-4445-9d46-f5625067652e admin admin] RBAC: Authorizing `identity:delete_project(project_id=3ea5113c49d4461695f17b1499fc7b43)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:33.064193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1bf1013-1bfe-4445-9d46-f5625067652e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:33.261196 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-d1bf1013-1bfe-4445-9d46-f5625067652e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3ea5113c49d4461695f17b1499fc7b43', 'request_id': 'req-d1bf1013-1bfe-4445-9d46-f5625067652e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:33.274763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-d1bf1013-1bfe-4445-9d46-f5625067652e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3ea5113c49d4461695f17b1499fc7b43', 'request_id': 'req-d1bf1013-1bfe-4445-9d46-f5625067652e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:33.295874 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-d1bf1013-1bfe-4445-9d46-f5625067652e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3ea5113c49d4461695f17b1499fc7b43', 'request_id': 'req-d1bf1013-1bfe-4445-9d46-f5625067652e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:33.471888 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae94ba3-6081-4737-843f-1c20b9c348cc service nova] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:33.473495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8ae94ba3-6081-4737-843f-1c20b9c348cc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:33.497559 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae94ba3-6081-4737-843f-1c20b9c348cc service nova] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:33.642247 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-d1bf1013-1bfe-4445-9d46-f5625067652e admin admin] The token cache is being invalidate because project 3ea5113c49d4461695f17b1499fc7b43 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:05:33.642247 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-d1bf1013-1bfe-4445-9d46-f5625067652e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:33.642247 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-d1bf1013-1bfe-4445-9d46-f5625067652e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:33.642247 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 913/1819] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:05:32 2022] DELETE /identity/v3/projects/3ea5113c49d4461695f17b1499fc7b43 => generated 0 bytes in 833 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:05:33.642247 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bf1013-1bfe-4445-9d46-f5625067652e admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:33.642684 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d1bf1013-1bfe-4445-9d46-f5625067652e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:33.649912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4167dd-daca-48bd-81b7-157e87038271 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:33.650558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4167dd-daca-48bd-81b7-157e87038271 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:33.650837 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4167dd-daca-48bd-81b7-157e87038271 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:33.651106 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4167dd-daca-48bd-81b7-157e87038271 admin admin] PATH_INFO: `/v3/projects/955511478fc2489e9f057dbf13bf3fc2` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:33.661918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4167dd-daca-48bd-81b7-157e87038271 admin admin] RBAC: Authorizing `identity:delete_project(project_id=955511478fc2489e9f057dbf13bf3fc2)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:33.664956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4167dd-daca-48bd-81b7-157e87038271 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:33.674462 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bf1013-1bfe-4445-9d46-f5625067652e admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:33.700228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb91aad-07ac-4128-b98d-a60d35888348 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:33.704209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb91aad-07ac-4128-b98d-a60d35888348 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:33.704209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb91aad-07ac-4128-b98d-a60d35888348 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:33.704209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb91aad-07ac-4128-b98d-a60d35888348 admin admin] PATH_INFO: `/v3/users/6ca095fb437447c4bdac0599d3783b17` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:33.710223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb91aad-07ac-4128-b98d-a60d35888348 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6ca095fb437447c4bdac0599d3783b17)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:33.712270 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb91aad-07ac-4128-b98d-a60d35888348 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:33.912780 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:33.995049 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8c4167dd-daca-48bd-81b7-157e87038271 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '955511478fc2489e9f057dbf13bf3fc2', 'request_id': 'req-8c4167dd-daca-48bd-81b7-157e87038271'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:34.025955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:INSERT=3,keystone:SELECT=90,keystone:DELETE=7 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:34.035438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8c4167dd-daca-48bd-81b7-157e87038271 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '955511478fc2489e9f057dbf13bf3fc2', 'request_id': 'req-8c4167dd-daca-48bd-81b7-157e87038271'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:34.103619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-deb91aad-07ac-4128-b98d-a60d35888348 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6ca095fb437447c4bdac0599d3783b17'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:34.152120 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8c4167dd-daca-48bd-81b7-157e87038271 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '955511478fc2489e9f057dbf13bf3fc2', 'request_id': 'req-8c4167dd-daca-48bd-81b7-157e87038271'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:34.181414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8c4167dd-daca-48bd-81b7-157e87038271 admin admin] The token cache is being invalidate because project 955511478fc2489e9f057dbf13bf3fc2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:05:34.182051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8c4167dd-daca-48bd-81b7-157e87038271 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:34.199545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8c4167dd-daca-48bd-81b7-157e87038271 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:34.205164 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 907/1820] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:05:33 2022] DELETE /identity/v3/projects/955511478fc2489e9f057dbf13bf3fc2 => generated 0 bytes in 734 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:34.263212 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4167dd-daca-48bd-81b7-157e87038271 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:34.266080 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8c4167dd-daca-48bd-81b7-157e87038271 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:34.453652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-deb91aad-07ac-4128-b98d-a60d35888348 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6ca095fb437447c4bdac0599d3783b17'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:34.504094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-deb91aad-07ac-4128-b98d-a60d35888348 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6ca095fb437447c4bdac0599d3783b17'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:34.504943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-deb91aad-07ac-4128-b98d-a60d35888348 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6ca095fb437447c4bdac0599d3783b17'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:34.548749 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-deb91aad-07ac-4128-b98d-a60d35888348 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6ca095fb437447c4bdac0599d3783b17'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:34.571283 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 914/1821] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:05:33 2022] DELETE /identity/v3/users/6ca095fb437447c4bdac0599d3783b17 => generated 0 bytes in 932 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:34.584187 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4167dd-daca-48bd-81b7-157e87038271 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:34.744214 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab4fb9e-d829-40f1-b1d2-40d3d2bda803 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:34.750061 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab4fb9e-d829-40f1-b1d2-40d3d2bda803 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:34.750350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab4fb9e-d829-40f1-b1d2-40d3d2bda803 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:34.750670 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab4fb9e-d829-40f1-b1d2-40d3d2bda803 admin admin] PATH_INFO: `/v3/users/d46cd6214aa34547855cec69962f0cfb` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:34.754000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab4fb9e-d829-40f1-b1d2-40d3d2bda803 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d46cd6214aa34547855cec69962f0cfb)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:34.764870 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab4fb9e-d829-40f1-b1d2-40d3d2bda803 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:34.778037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7809c9ea-2f63-4b08-ab66-353158e15755 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:34.778319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7809c9ea-2f63-4b08-ab66-353158e15755 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:34.778784 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7809c9ea-2f63-4b08-ab66-353158e15755 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:34.781247 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 915/1822] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 4 01:05:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:05:35.011646 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9957feca-47fc-4c2f-90fc-4a2a6fc7063e None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:35.011646 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9957feca-47fc-4c2f-90fc-4a2a6fc7063e None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:35.011646 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9957feca-47fc-4c2f-90fc-4a2a6fc7063e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:35.011646 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9957feca-47fc-4c2f-90fc-4a2a6fc7063e None None] MFA Rules not processed for user `57c88f16c7df40adb917f49804fda21f`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:35.011646 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9957feca-47fc-4c2f-90fc-4a2a6fc7063e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:35.148887 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 916/1823] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 4 01:05:34 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:05:35.206117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9957feca-47fc-4c2f-90fc-4a2a6fc7063e None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:35.210059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9957feca-47fc-4c2f-90fc-4a2a6fc7063e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:35.263302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9957feca-47fc-4c2f-90fc-4a2a6fc7063e None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:35.293939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f37271-dc50-48d5-8684-b363c4e1a8d7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:35.294788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f37271-dc50-48d5-8684-b363c4e1a8d7 service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:35.295076 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f37271-dc50-48d5-8684-b363c4e1a8d7 service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:35.295350 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f37271-dc50-48d5-8684-b363c4e1a8d7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:35.328143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f37271-dc50-48d5-8684-b363c4e1a8d7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:35.329630 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f37271-dc50-48d5-8684-b363c4e1a8d7 service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:35.359037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 917/1824] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:35.574669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7ab4fb9e-d829-40f1-b1d2-40d3d2bda803 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd46cd6214aa34547855cec69962f0cfb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:35.576734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7ab4fb9e-d829-40f1-b1d2-40d3d2bda803 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd46cd6214aa34547855cec69962f0cfb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:35.717554 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7ab4fb9e-d829-40f1-b1d2-40d3d2bda803 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd46cd6214aa34547855cec69962f0cfb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:35.748214 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7ab4fb9e-d829-40f1-b1d2-40d3d2bda803 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd46cd6214aa34547855cec69962f0cfb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:35.752314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7ab4fb9e-d829-40f1-b1d2-40d3d2bda803 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd46cd6214aa34547855cec69962f0cfb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:35.790799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 908/1825] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:05:34 2022] DELETE /identity/v3/users/d46cd6214aa34547855cec69962f0cfb => generated 0 bytes in 1529 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:05:36.905120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f37271-dc50-48d5-8684-b363c4e1a8d7 service nova] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:36.907413 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-42f37271-dc50-48d5-8684-b363c4e1a8d7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:36.941236 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f37271-dc50-48d5-8684-b363c4e1a8d7 service nova] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:37.029977 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661e85bf-8058-4216-b64a-24395ffd725d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:37.031068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661e85bf-8058-4216-b64a-24395ffd725d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:37.032941 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661e85bf-8058-4216-b64a-24395ffd725d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:37.033603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661e85bf-8058-4216-b64a-24395ffd725d admin admin] PATH_INFO: `/v3/projects/42233f43b574453eb16ec7c6c7437905` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:37.036549 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661e85bf-8058-4216-b64a-24395ffd725d admin admin] RBAC: Authorizing `identity:delete_project(project_id=42233f43b574453eb16ec7c6c7437905)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:37.039432 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661e85bf-8058-4216-b64a-24395ffd725d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:37.141756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:37.162033 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-661e85bf-8058-4216-b64a-24395ffd725d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '42233f43b574453eb16ec7c6c7437905', 'request_id': 'req-661e85bf-8058-4216-b64a-24395ffd725d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:37.167619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-661e85bf-8058-4216-b64a-24395ffd725d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '42233f43b574453eb16ec7c6c7437905', 'request_id': 'req-661e85bf-8058-4216-b64a-24395ffd725d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:37.174127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-661e85bf-8058-4216-b64a-24395ffd725d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '42233f43b574453eb16ec7c6c7437905', 'request_id': 'req-661e85bf-8058-4216-b64a-24395ffd725d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:37.207153 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab4fb9e-d829-40f1-b1d2-40d3d2bda803 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:37.209223 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-7ab4fb9e-d829-40f1-b1d2-40d3d2bda803 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:37.226553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab4fb9e-d829-40f1-b1d2-40d3d2bda803 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:37.275433 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-661e85bf-8058-4216-b64a-24395ffd725d admin admin] The token cache is being invalidate because project 42233f43b574453eb16ec7c6c7437905 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:05:37.275834 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-661e85bf-8058-4216-b64a-24395ffd725d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:37.276976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-661e85bf-8058-4216-b64a-24395ffd725d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:37.279154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 918/1826] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:05:36 2022] DELETE /identity/v3/projects/42233f43b574453eb16ec7c6c7437905 => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:37.284799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bc022d-244a-452b-9bdc-710b99bc9736 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:37.285170 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bc022d-244a-452b-9bdc-710b99bc9736 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:37.285661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bc022d-244a-452b-9bdc-710b99bc9736 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:37.296766 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3aea181-f68c-49d8-8764-342c7f33645a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:37.297769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3aea181-f68c-49d8-8764-342c7f33645a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:37.298518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3aea181-f68c-49d8-8764-342c7f33645a admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:37.299206 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3aea181-f68c-49d8-8764-342c7f33645a admin admin] PATH_INFO: `/v3/projects/d73d808e831f4d1a982b7f538a5853de` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:37.301257 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3aea181-f68c-49d8-8764-342c7f33645a admin admin] RBAC: Authorizing `identity:delete_project(project_id=d73d808e831f4d1a982b7f538a5853de)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:37.303374 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3aea181-f68c-49d8-8764-342c7f33645a admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:37.318868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-a0bc022d-244a-452b-9bdc-710b99bc9736 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:37.318868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a0bc022d-244a-452b-9bdc-710b99bc9736 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:37.345166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 919/1827] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:37.359420 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b67790-3243-48d3-9a55-38c7ea0fbf5a None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:37.359823 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b67790-3243-48d3-9a55-38c7ea0fbf5a None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:37.360103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b67790-3243-48d3-9a55-38c7ea0fbf5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:37.380392 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-62b67790-3243-48d3-9a55-38c7ea0fbf5a None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:37.381623 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-62b67790-3243-48d3-9a55-38c7ea0fbf5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:37.389487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 920/1828] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:05:37.394153 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b67790-3243-48d3-9a55-38c7ea0fbf5a None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:37.403201 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b67790-3243-48d3-9a55-38c7ea0fbf5a None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:37.407774 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca95f8fc-a4b2-4bd5-bb9f-ecf0f58137d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:37.410991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca95f8fc-a4b2-4bd5-bb9f-ecf0f58137d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:37.410991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca95f8fc-a4b2-4bd5-bb9f-ecf0f58137d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:37.410991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca95f8fc-a4b2-4bd5-bb9f-ecf0f58137d3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:37.410991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca95f8fc-a4b2-4bd5-bb9f-ecf0f58137d3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:05:37.410991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca95f8fc-a4b2-4bd5-bb9f-ecf0f58137d3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:37.412195 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca95f8fc-a4b2-4bd5-bb9f-ecf0f58137d3 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:37.423468 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 921/1829] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:05:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 01:05:37.427842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca95f8fc-a4b2-4bd5-bb9f-ecf0f58137d3 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:37.437730 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca95f8fc-a4b2-4bd5-bb9f-ecf0f58137d3 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:37.448469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d3aea181-f68c-49d8-8764-342c7f33645a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd73d808e831f4d1a982b7f538a5853de', 'request_id': 'req-d3aea181-f68c-49d8-8764-342c7f33645a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:37.454099 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d3aea181-f68c-49d8-8764-342c7f33645a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd73d808e831f4d1a982b7f538a5853de', 'request_id': 'req-d3aea181-f68c-49d8-8764-342c7f33645a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:37.499594 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c7dd51-8e26-4216-be30-6ce7facf67ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:37.500470 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c7dd51-8e26-4216-be30-6ce7facf67ab admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:37.500614 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c7dd51-8e26-4216-be30-6ce7facf67ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:37.500896 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c7dd51-8e26-4216-be30-6ce7facf67ab admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:37.501566 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c7dd51-8e26-4216-be30-6ce7facf67ab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:37.503642 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c7dd51-8e26-4216-be30-6ce7facf67ab admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:37.568751 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d3aea181-f68c-49d8-8764-342c7f33645a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd73d808e831f4d1a982b7f538a5853de', 'request_id': 'req-d3aea181-f68c-49d8-8764-342c7f33645a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:37.583820 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d3aea181-f68c-49d8-8764-342c7f33645a admin admin] The token cache is being invalidate because project d73d808e831f4d1a982b7f538a5853de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:05:37.584414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d3aea181-f68c-49d8-8764-342c7f33645a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:37.585351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d3aea181-f68c-49d8-8764-342c7f33645a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:37.587267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 909/1830] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:05:37 2022] DELETE /identity/v3/projects/d73d808e831f4d1a982b7f538a5853de => generated 0 bytes in 381 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:05:37.592019 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db518da-61ec-45ee-92db-70c76375e680 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:37.592455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db518da-61ec-45ee-92db-70c76375e680 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:37.592738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db518da-61ec-45ee-92db-70c76375e680 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:37.606493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 922/1831] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:37 2022] POST /identity/v3/projects => generated 375 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:05:37.610833 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c7dd51-8e26-4216-be30-6ce7facf67ab admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:37.613681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c6c7dd51-8e26-4216-be30-6ce7facf67ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:37.620705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-8db518da-61ec-45ee-92db-70c76375e680 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:37.626844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8db518da-61ec-45ee-92db-70c76375e680 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:37.700624 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 910/1832] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:05:37.707612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c7dd51-8e26-4216-be30-6ce7facf67ab admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:37.709730 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52df57af-490d-406d-bf3d-706be999b43d None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:37.715245 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52df57af-490d-406d-bf3d-706be999b43d None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:37.715245 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52df57af-490d-406d-bf3d-706be999b43d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:37.715450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a932894-b8dd-4cfa-a042-bfc3852eaea8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:37.715450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a932894-b8dd-4cfa-a042-bfc3852eaea8 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:37.715450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a932894-b8dd-4cfa-a042-bfc3852eaea8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:37.715450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a932894-b8dd-4cfa-a042-bfc3852eaea8 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:37.716020 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a932894-b8dd-4cfa-a042-bfc3852eaea8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:37.718321 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a932894-b8dd-4cfa-a042-bfc3852eaea8 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:37.733579 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-52df57af-490d-406d-bf3d-706be999b43d None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:37.734841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-52df57af-490d-406d-bf3d-706be999b43d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:37.745016 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 911/1833] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:37.749288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52df57af-490d-406d-bf3d-706be999b43d None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:37.759812 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52df57af-490d-406d-bf3d-706be999b43d None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:37.766300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6f1662-d03b-4d5c-8225-bb5cb1a103d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:37.766937 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6f1662-d03b-4d5c-8225-bb5cb1a103d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:37.767219 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6f1662-d03b-4d5c-8225-bb5cb1a103d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:37.767490 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6f1662-d03b-4d5c-8225-bb5cb1a103d1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:37.768153 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6f1662-d03b-4d5c-8225-bb5cb1a103d1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:05:37.769166 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6f1662-d03b-4d5c-8225-bb5cb1a103d1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:37.770772 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6f1662-d03b-4d5c-8225-bb5cb1a103d1 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:37.782597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 912/1834] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:05:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 01:05:37.787523 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6f1662-d03b-4d5c-8225-bb5cb1a103d1 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:37.792714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 923/1835] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:37 2022] POST /identity/v3/users => generated 288 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:37.804247 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6f1662-d03b-4d5c-8225-bb5cb1a103d1 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:37.805144 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a932894-b8dd-4cfa-a042-bfc3852eaea8 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:37.810283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb59b62-92fc-433c-ae81-682dde043591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:37.810985 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb59b62-92fc-433c-ae81-682dde043591 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:37.811321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb59b62-92fc-433c-ae81-682dde043591 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:37.811810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb59b62-92fc-433c-ae81-682dde043591 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:37.812473 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb59b62-92fc-433c-ae81-682dde043591 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:37.815715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb59b62-92fc-433c-ae81-682dde043591 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:37.818980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a932894-b8dd-4cfa-a042-bfc3852eaea8 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:37.834774 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa218b1-fea0-48aa-9e10-4a46f9bb34fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:37.836845 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa218b1-fea0-48aa-9e10-4a46f9bb34fb admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:37.837347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa218b1-fea0-48aa-9e10-4a46f9bb34fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:37.838622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa218b1-fea0-48aa-9e10-4a46f9bb34fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:37.839507 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa218b1-fea0-48aa-9e10-4a46f9bb34fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:37.841990 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa218b1-fea0-48aa-9e10-4a46f9bb34fb admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:37.860085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 924/1836] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:37 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:05:37.860085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa218b1-fea0-48aa-9e10-4a46f9bb34fb admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:37.869197 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa218b1-fea0-48aa-9e10-4a46f9bb34fb admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:37.873978 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2458d097-6ef1-46ec-99ca-e63bb4617272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:37.874584 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2458d097-6ef1-46ec-99ca-e63bb4617272 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:37.874958 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2458d097-6ef1-46ec-99ca-e63bb4617272 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:37.875238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2458d097-6ef1-46ec-99ca-e63bb4617272 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:37.875885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2458d097-6ef1-46ec-99ca-e63bb4617272 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:37.877579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2458d097-6ef1-46ec-99ca-e63bb4617272 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:37.889009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 925/1837] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:37 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:05:37.893248 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2458d097-6ef1-46ec-99ca-e63bb4617272 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:37.903663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2458d097-6ef1-46ec-99ca-e63bb4617272 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:37.908746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd687d3-37f8-4b19-8bd6-fe6874aa65c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:37.910378 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd687d3-37f8-4b19-8bd6-fe6874aa65c8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:37.910899 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd687d3-37f8-4b19-8bd6-fe6874aa65c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:37.911383 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd687d3-37f8-4b19-8bd6-fe6874aa65c8 admin admin] PATH_INFO: `/v3/projects/7eb1a3a92ce841f4a6955bc13a3bc6e0/users/a5abc0e087ec4e13bed1d23a347f23b2/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:37.915173 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 913/1838] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:37 2022] POST /identity/v3/projects => generated 387 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:37.916401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb59b62-92fc-433c-ae81-682dde043591 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:37.932146 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd687d3-37f8-4b19-8bd6-fe6874aa65c8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7eb1a3a92ce841f4a6955bc13a3bc6e0, user_id=a5abc0e087ec4e13bed1d23a347f23b2, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:37.932575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd687d3-37f8-4b19-8bd6-fe6874aa65c8 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:37.979921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb59b62-92fc-433c-ae81-682dde043591 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:37.984682 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d05d81-cf00-4921-9125-c7c9f60f027b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:37.985323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d05d81-cf00-4921-9125-c7c9f60f027b admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:37.985610 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d05d81-cf00-4921-9125-c7c9f60f027b admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:37.985886 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d05d81-cf00-4921-9125-c7c9f60f027b admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:37.986785 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d05d81-cf00-4921-9125-c7c9f60f027b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:37.989649 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d05d81-cf00-4921-9125-c7c9f60f027b admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:37.992061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 926/1839] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:37 2022] PUT /identity/v3/projects/7eb1a3a92ce841f4a6955bc13a3bc6e0/users/a5abc0e087ec4e13bed1d23a347f23b2/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:38.001276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dfaa5f-4db1-4ddc-b115-f487d9c1d246 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:38.001564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dfaa5f-4db1-4ddc-b115-f487d9c1d246 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:38.001840 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dfaa5f-4db1-4ddc-b115-f487d9c1d246 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:38.030325 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-20dfaa5f-4db1-4ddc-b115-f487d9c1d246 None None] MFA Rules not processed for user `a5abc0e087ec4e13bed1d23a347f23b2`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:38.031544 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-20dfaa5f-4db1-4ddc-b115-f487d9c1d246 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:38.076053 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 914/1840] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:37 2022] POST /identity/v3/users => generated 294 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:38.082621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d05d81-cf00-4921-9125-c7c9f60f027b admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:38.092357 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d05d81-cf00-4921-9125-c7c9f60f027b admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:38.099764 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 927/1841] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:37 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:05:38.112867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20dfaa5f-4db1-4ddc-b115-f487d9c1d246 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:38.114331 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-20dfaa5f-4db1-4ddc-b115-f487d9c1d246 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:38.150684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f60b6ec-81f3-4fc4-a99b-d61077644c28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:38.151502 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f60b6ec-81f3-4fc4-a99b-d61077644c28 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:38.153214 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:38.153721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f60b6ec-81f3-4fc4-a99b-d61077644c28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:38.154465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f60b6ec-81f3-4fc4-a99b-d61077644c28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:38.155217 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f60b6ec-81f3-4fc4-a99b-d61077644c28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:38.157334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f60b6ec-81f3-4fc4-a99b-d61077644c28 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:38.168946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20dfaa5f-4db1-4ddc-b115-f487d9c1d246 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:38.169929 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 915/1842] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:38 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:05:38.172528 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f0c4d7-ea72-42e6-b5e6-db8b70581539 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:38.173163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f0c4d7-ea72-42e6-b5e6-db8b70581539 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:38.173448 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f0c4d7-ea72-42e6-b5e6-db8b70581539 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:38.173731 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f0c4d7-ea72-42e6-b5e6-db8b70581539 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:38.174094 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f60b6ec-81f3-4fc4-a99b-d61077644c28 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:38.183581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f0c4d7-ea72-42e6-b5e6-db8b70581539 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:38.186045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f0c4d7-ea72-42e6-b5e6-db8b70581539 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:38.186559 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f60b6ec-81f3-4fc4-a99b-d61077644c28 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:38.192352 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2dff62-aa53-41a6-b865-9e8b57fc7665 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:38.193137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2dff62-aa53-41a6-b865-9e8b57fc7665 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:38.193428 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2dff62-aa53-41a6-b865-9e8b57fc7665 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:38.193693 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2dff62-aa53-41a6-b865-9e8b57fc7665 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:38.194337 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2dff62-aa53-41a6-b865-9e8b57fc7665 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:38.195118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 928/1843] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:38 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:05:38.197421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2dff62-aa53-41a6-b865-9e8b57fc7665 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:38.209517 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 916/1844] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:38 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:05:38.212743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f0c4d7-ea72-42e6-b5e6-db8b70581539 service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:38.224165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f0c4d7-ea72-42e6-b5e6-db8b70581539 service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:38.229118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444d8fcf-4e4e-4776-925a-a0af2d3c91bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:38.229855 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444d8fcf-4e4e-4776-925a-a0af2d3c91bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:38.230136 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444d8fcf-4e4e-4776-925a-a0af2d3c91bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:38.230403 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444d8fcf-4e4e-4776-925a-a0af2d3c91bb admin admin] PATH_INFO: `/v3/projects/8d0a8304bc0845249f776ea53f3567cf/users/ee20b7bc96e940a0b6fd7dd8284270e9/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:38.249456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444d8fcf-4e4e-4776-925a-a0af2d3c91bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d0a8304bc0845249f776ea53f3567cf, user_id=ee20b7bc96e940a0b6fd7dd8284270e9, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:38.252196 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444d8fcf-4e4e-4776-925a-a0af2d3c91bb admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:38.271965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:38.295317 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 929/1845] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:38 2022] PUT /identity/v3/projects/8d0a8304bc0845249f776ea53f3567cf/users/ee20b7bc96e940a0b6fd7dd8284270e9/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:05:38.306366 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a93113-9aa2-4a65-9c4f-df70104c9eac None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:38.306659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a93113-9aa2-4a65-9c4f-df70104c9eac None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:38.306957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a93113-9aa2-4a65-9c4f-df70104c9eac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:38.337397 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-82a93113-9aa2-4a65-9c4f-df70104c9eac None None] MFA Rules not processed for user `ee20b7bc96e940a0b6fd7dd8284270e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:38.338612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-82a93113-9aa2-4a65-9c4f-df70104c9eac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:38.412532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 917/1846] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:38 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:38.417052 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444d8fcf-4e4e-4776-925a-a0af2d3c91bb admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:38.495941 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444d8fcf-4e4e-4776-925a-a0af2d3c91bb admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:38.502457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c633f0-0f5c-4828-b90e-f7a4af3dff29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:38.507220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c633f0-0f5c-4828-b90e-f7a4af3dff29 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:38.507220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c633f0-0f5c-4828-b90e-f7a4af3dff29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:38.507220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c633f0-0f5c-4828-b90e-f7a4af3dff29 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:38.507220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c633f0-0f5c-4828-b90e-f7a4af3dff29 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:38.507220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c633f0-0f5c-4828-b90e-f7a4af3dff29 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:38.627319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 930/1847] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:38 2022] POST /identity/v3/projects => generated 387 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:05:38.632318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a93113-9aa2-4a65-9c4f-df70104c9eac None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:38.643061 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a93113-9aa2-4a65-9c4f-df70104c9eac None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:38.700184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75e2246-aa11-4fd2-a654-30bfca7ac703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:38.700794 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75e2246-aa11-4fd2-a654-30bfca7ac703 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:38.701140 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75e2246-aa11-4fd2-a654-30bfca7ac703 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:38.701424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75e2246-aa11-4fd2-a654-30bfca7ac703 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:38.702024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75e2246-aa11-4fd2-a654-30bfca7ac703 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:38.704029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75e2246-aa11-4fd2-a654-30bfca7ac703 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:38.820907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 918/1848] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:38 2022] POST /identity/v3/users => generated 294 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:38.825141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c633f0-0f5c-4828-b90e-f7a4af3dff29 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:38.835742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c633f0-0f5c-4828-b90e-f7a4af3dff29 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:38.840249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12efb73-3146-4e1f-9b18-669dfb3caacb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:38.840866 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12efb73-3146-4e1f-9b18-669dfb3caacb admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:38.842491 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12efb73-3146-4e1f-9b18-669dfb3caacb admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:38.843053 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12efb73-3146-4e1f-9b18-669dfb3caacb admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:38.843914 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12efb73-3146-4e1f-9b18-669dfb3caacb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:38.846117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f12efb73-3146-4e1f-9b18-669dfb3caacb admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:38.864028 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 931/1849] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:38 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:38.865330 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75e2246-aa11-4fd2-a654-30bfca7ac703 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:38.875725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75e2246-aa11-4fd2-a654-30bfca7ac703 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:38.881899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67739d6c-3ead-4a04-a292-a44f82867ce0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:38.881899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67739d6c-3ead-4a04-a292-a44f82867ce0 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:38.882379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67739d6c-3ead-4a04-a292-a44f82867ce0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:38.882465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67739d6c-3ead-4a04-a292-a44f82867ce0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:38.883123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67739d6c-3ead-4a04-a292-a44f82867ce0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:38.884747 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67739d6c-3ead-4a04-a292-a44f82867ce0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:38.898381 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 919/1850] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:38 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:38.899589 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12efb73-3146-4e1f-9b18-669dfb3caacb admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:38.910110 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f12efb73-3146-4e1f-9b18-669dfb3caacb admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:38.914762 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff83b494-d269-4528-aba5-3fa679b7fcdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:38.915536 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff83b494-d269-4528-aba5-3fa679b7fcdb admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:38.915814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff83b494-d269-4528-aba5-3fa679b7fcdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:38.916088 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff83b494-d269-4528-aba5-3fa679b7fcdb admin admin] PATH_INFO: `/v3/projects/174be28425554302a7e0829820fda15b/users/d92b5fa9dc9d4358beac4808a070d9be/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:38.931481 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff83b494-d269-4528-aba5-3fa679b7fcdb admin admin] RBAC: Authorizing `identity:create_grant(project_id=174be28425554302a7e0829820fda15b, user_id=d92b5fa9dc9d4358beac4808a070d9be, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:38.934261 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff83b494-d269-4528-aba5-3fa679b7fcdb admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:39.006534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 932/1851] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:38 2022] PUT /identity/v3/projects/174be28425554302a7e0829820fda15b/users/d92b5fa9dc9d4358beac4808a070d9be/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:39.017929 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bb97d5-8689-4b6d-b26f-ce429b123818 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:39.018244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bb97d5-8689-4b6d-b26f-ce429b123818 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:39.018529 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bb97d5-8689-4b6d-b26f-ce429b123818 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:39.046996 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-f3bb97d5-8689-4b6d-b26f-ce429b123818 None None] MFA Rules not processed for user `d92b5fa9dc9d4358beac4808a070d9be`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:39.048265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f3bb97d5-8689-4b6d-b26f-ce429b123818 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:39.124356 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 920/1852] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:39 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:39.128885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff83b494-d269-4528-aba5-3fa679b7fcdb admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:39.192126 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff83b494-d269-4528-aba5-3fa679b7fcdb admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:39.196750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaeb6270-6a2c-4b40-a405-039541064f2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:39.197462 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaeb6270-6a2c-4b40-a405-039541064f2e admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:39.197744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaeb6270-6a2c-4b40-a405-039541064f2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:39.198020 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaeb6270-6a2c-4b40-a405-039541064f2e admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:39.198654 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaeb6270-6a2c-4b40-a405-039541064f2e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:39.200711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaeb6270-6a2c-4b40-a405-039541064f2e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:39.255258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:39.293833 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 933/1853] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:39 2022] POST /identity/v3/projects => generated 385 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:39.298176 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bb97d5-8689-4b6d-b26f-ce429b123818 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:39.313310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bb97d5-8689-4b6d-b26f-ce429b123818 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:39.365943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:DELETE=1,keystone:SELECT=95,keystone:INSERT=4 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:39.379087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332be57c-8c0c-42c6-8cad-d8d0254d5800 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:39.379694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332be57c-8c0c-42c6-8cad-d8d0254d5800 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:39.379973 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332be57c-8c0c-42c6-8cad-d8d0254d5800 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:39.380244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332be57c-8c0c-42c6-8cad-d8d0254d5800 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:39.380878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332be57c-8c0c-42c6-8cad-d8d0254d5800 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:39.382880 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332be57c-8c0c-42c6-8cad-d8d0254d5800 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:39.504076 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 921/1854] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:39 2022] POST /identity/v3/users => generated 299 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:05:39.505415 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaeb6270-6a2c-4b40-a405-039541064f2e admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:39.528629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaeb6270-6a2c-4b40-a405-039541064f2e admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:39.558228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e27b82-7d7e-4d24-a818-abd084c71649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:39.558984 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e27b82-7d7e-4d24-a818-abd084c71649 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:39.559330 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e27b82-7d7e-4d24-a818-abd084c71649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:39.559665 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e27b82-7d7e-4d24-a818-abd084c71649 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:39.560415 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e27b82-7d7e-4d24-a818-abd084c71649 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:39.562185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e27b82-7d7e-4d24-a818-abd084c71649 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:39.590981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 934/1855] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:39 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:05:39.592774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332be57c-8c0c-42c6-8cad-d8d0254d5800 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:39.606360 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332be57c-8c0c-42c6-8cad-d8d0254d5800 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:39.611215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cdee12-b172-4046-845b-594704602900 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:39.611878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cdee12-b172-4046-845b-594704602900 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:39.612357 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cdee12-b172-4046-845b-594704602900 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:39.612738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6cdee12-b172-4046-845b-594704602900 admin admin] PATH_INFO: `/v3/projects/760186835723424d857dde933c062bea/users/18f656bdeb014871b71b138280616beb/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:39.633399 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cdee12-b172-4046-845b-594704602900 admin admin] RBAC: Authorizing `identity:create_grant(project_id=760186835723424d857dde933c062bea, user_id=18f656bdeb014871b71b138280616beb, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:39.637220 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6cdee12-b172-4046-845b-594704602900 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:39.711483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 922/1856] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:39 2022] PUT /identity/v3/projects/760186835723424d857dde933c062bea/users/18f656bdeb014871b71b138280616beb/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:39.715951 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e27b82-7d7e-4d24-a818-abd084c71649 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:39.736750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e27b82-7d7e-4d24-a818-abd084c71649 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:39.794084 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600a4c6d-2d23-460e-bab7-3776293ff70e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:39.794689 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600a4c6d-2d23-460e-bab7-3776293ff70e admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:39.794969 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600a4c6d-2d23-460e-bab7-3776293ff70e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:39.795251 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600a4c6d-2d23-460e-bab7-3776293ff70e admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:39.795952 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600a4c6d-2d23-460e-bab7-3776293ff70e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:39.797625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600a4c6d-2d23-460e-bab7-3776293ff70e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:39.811360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 935/1857] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:39 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:39.812527 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cdee12-b172-4046-845b-594704602900 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:39.823738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6cdee12-b172-4046-845b-594704602900 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:39.828660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a45a908-590f-4e8f-a58f-3aad98d408bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:39.829358 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a45a908-590f-4e8f-a58f-3aad98d408bc admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:39.829641 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a45a908-590f-4e8f-a58f-3aad98d408bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:39.829912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a45a908-590f-4e8f-a58f-3aad98d408bc admin admin] PATH_INFO: `/v3/projects/760186835723424d857dde933c062bea/users/18f656bdeb014871b71b138280616beb/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:39.832751 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a45a908-590f-4e8f-a58f-3aad98d408bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=760186835723424d857dde933c062bea, user_id=18f656bdeb014871b71b138280616beb, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:39.835433 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a45a908-590f-4e8f-a58f-3aad98d408bc admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:39.847590 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 923/1858] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:39 2022] PUT /identity/v3/projects/760186835723424d857dde933c062bea/users/18f656bdeb014871b71b138280616beb/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:05:39.857970 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba805562-c20e-46a9-b74c-a109c41f34fd None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:39.858265 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba805562-c20e-46a9-b74c-a109c41f34fd None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:39.858545 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba805562-c20e-46a9-b74c-a109c41f34fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:39.887743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-ba805562-c20e-46a9-b74c-a109c41f34fd None None] MFA Rules not processed for user `18f656bdeb014871b71b138280616beb`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:39.891397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ba805562-c20e-46a9-b74c-a109c41f34fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:39.969402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 936/1859] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:39 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:39.985208 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a45a908-590f-4e8f-a58f-3aad98d408bc admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:40.046182 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a45a908-590f-4e8f-a58f-3aad98d408bc admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:40.050789 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106d006c-e77f-4e81-af5c-b406284670ac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:40.051215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106d006c-e77f-4e81-af5c-b406284670ac service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:40.051498 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106d006c-e77f-4e81-af5c-b406284670ac service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:40.051772 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106d006c-e77f-4e81-af5c-b406284670ac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:40.060543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106d006c-e77f-4e81-af5c-b406284670ac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:40.062030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106d006c-e77f-4e81-af5c-b406284670ac service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:40.071624 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 924/1860] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:39 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:40.181837 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba805562-c20e-46a9-b74c-a109c41f34fd None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:40.183618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ba805562-c20e-46a9-b74c-a109c41f34fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:40.210692 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba805562-c20e-46a9-b74c-a109c41f34fd None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:40.214483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa959a38-0c08-4887-b010-a2a772a08f5b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:40.215293 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa959a38-0c08-4887-b010-a2a772a08f5b service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:40.215718 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa959a38-0c08-4887-b010-a2a772a08f5b service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:40.216155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa959a38-0c08-4887-b010-a2a772a08f5b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:40.219307 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-fa959a38-0c08-4887-b010-a2a772a08f5b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:40.294136 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa959a38-0c08-4887-b010-a2a772a08f5b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:40.295843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa959a38-0c08-4887-b010-a2a772a08f5b service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:40.306559 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 937/1861] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:40 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:40.578717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:40.883378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:41.531963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:41.684960 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106d006c-e77f-4e81-af5c-b406284670ac service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:41.694403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106d006c-e77f-4e81-af5c-b406284670ac service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:41.698096 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2584555e-ed29-4459-bbdd-1bef526ceec3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:41.698958 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2584555e-ed29-4459-bbdd-1bef526ceec3 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:41.699239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2584555e-ed29-4459-bbdd-1bef526ceec3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:41.699504 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2584555e-ed29-4459-bbdd-1bef526ceec3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:41.767388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2584555e-ed29-4459-bbdd-1bef526ceec3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:41.768850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2584555e-ed29-4459-bbdd-1bef526ceec3 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:41.777890 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 925/1862] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:41 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:41.797256 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:42.910442 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:42.978776 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:43.806869 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:43.875639 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:44.408909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:44.438804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:44.749721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:INSERT=7,keystone:SELECT=93 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:44.785946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:46.293178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa959a38-0c08-4887-b010-a2a772a08f5b service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:46.386796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:46.400374 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa959a38-0c08-4887-b010-a2a772a08f5b service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:46.406555 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d2c807-9fa6-438e-8fb0-b2e621e7f5df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:46.408007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d2c807-9fa6-438e-8fb0-b2e621e7f5df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:46.408288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d2c807-9fa6-438e-8fb0-b2e621e7f5df admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:46.409999 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d2c807-9fa6-438e-8fb0-b2e621e7f5df admin admin] PATH_INFO: `/v3/users/a5abc0e087ec4e13bed1d23a347f23b2` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:46.411744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d2c807-9fa6-438e-8fb0-b2e621e7f5df admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5abc0e087ec4e13bed1d23a347f23b2)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:46.413788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d2c807-9fa6-438e-8fb0-b2e621e7f5df admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:46.622387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-87d2c807-9fa6-438e-8fb0-b2e621e7f5df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5abc0e087ec4e13bed1d23a347f23b2'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:46.688456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-87d2c807-9fa6-438e-8fb0-b2e621e7f5df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5abc0e087ec4e13bed1d23a347f23b2'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:46.713788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-87d2c807-9fa6-438e-8fb0-b2e621e7f5df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5abc0e087ec4e13bed1d23a347f23b2'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:46.714886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-87d2c807-9fa6-438e-8fb0-b2e621e7f5df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5abc0e087ec4e13bed1d23a347f23b2'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:46.729709 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-87d2c807-9fa6-438e-8fb0-b2e621e7f5df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5abc0e087ec4e13bed1d23a347f23b2'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:46.737103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 938/1863] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:05:46 2022] DELETE /identity/v3/users/a5abc0e087ec4e13bed1d23a347f23b2 => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:46.742044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223be669-d3b8-4709-b84d-04403aee2498 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:46.742896 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223be669-d3b8-4709-b84d-04403aee2498 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:46.743470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223be669-d3b8-4709-b84d-04403aee2498 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:46.764873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-223be669-d3b8-4709-b84d-04403aee2498 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:46.766247 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-223be669-d3b8-4709-b84d-04403aee2498 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:46.835272 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 926/1864] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:46.864681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d2c807-9fa6-438e-8fb0-b2e621e7f5df admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:46.866289 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-87d2c807-9fa6-438e-8fb0-b2e621e7f5df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:46.947712 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d2c807-9fa6-438e-8fb0-b2e621e7f5df admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:46.956351 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306205c6-febb-411b-b263-eb8df5fff6c1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:46.956351 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306205c6-febb-411b-b263-eb8df5fff6c1 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:46.956351 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306205c6-febb-411b-b263-eb8df5fff6c1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:46.956351 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306205c6-febb-411b-b263-eb8df5fff6c1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:46.968927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306205c6-febb-411b-b263-eb8df5fff6c1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:46.970725 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306205c6-febb-411b-b263-eb8df5fff6c1 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:46.990501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 939/1865] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:47.881948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785fc928-5567-48af-ba81-1c5b8e6b962c None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:47.882514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785fc928-5567-48af-ba81-1c5b8e6b962c None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:47.882514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785fc928-5567-48af-ba81-1c5b8e6b962c None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:47.884475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 927/1866] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 4 01:05:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:05:47.932786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b48b93-f3de-4d9a-8e3b-ff9472da6b8a None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:47.933376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b48b93-f3de-4d9a-8e3b-ff9472da6b8a None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:47.933797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b48b93-f3de-4d9a-8e3b-ff9472da6b8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:47.961148 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-49b48b93-f3de-4d9a-8e3b-ff9472da6b8a None None] MFA Rules not processed for user `57c88f16c7df40adb917f49804fda21f`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:47.962489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-49b48b93-f3de-4d9a-8e3b-ff9472da6b8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:48.112223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 940/1867] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 4 01:05:47 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:05:48.115299 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785fc928-5567-48af-ba81-1c5b8e6b962c None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:48.117165 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-785fc928-5567-48af-ba81-1c5b8e6b962c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:48.151888 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785fc928-5567-48af-ba81-1c5b8e6b962c None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:48.170779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a8d26d-39ad-4bc4-9e1f-cd8b37ed4331 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:48.171625 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a8d26d-39ad-4bc4-9e1f-cd8b37ed4331 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:48.172007 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a8d26d-39ad-4bc4-9e1f-cd8b37ed4331 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:48.172379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a8d26d-39ad-4bc4-9e1f-cd8b37ed4331 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:48.183288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a8d26d-39ad-4bc4-9e1f-cd8b37ed4331 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:48.185400 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a8d26d-39ad-4bc4-9e1f-cd8b37ed4331 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:48.198829 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 928/1868] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:48 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:48.706404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b48b93-f3de-4d9a-8e3b-ff9472da6b8a None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:48.707906 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-49b48b93-f3de-4d9a-8e3b-ff9472da6b8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:48.723290 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b48b93-f3de-4d9a-8e3b-ff9472da6b8a None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:48.729135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693acbf5-4dc3-40cc-924f-edf5fb0ea2a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:48.729135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693acbf5-4dc3-40cc-924f-edf5fb0ea2a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:48.729674 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693acbf5-4dc3-40cc-924f-edf5fb0ea2a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:48.729968 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693acbf5-4dc3-40cc-924f-edf5fb0ea2a7 admin admin] PATH_INFO: `/v3/projects/7eb1a3a92ce841f4a6955bc13a3bc6e0` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:48.731269 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693acbf5-4dc3-40cc-924f-edf5fb0ea2a7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7eb1a3a92ce841f4a6955bc13a3bc6e0)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:48.733728 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693acbf5-4dc3-40cc-924f-edf5fb0ea2a7 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:48.737875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:48.878858 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-693acbf5-4dc3-40cc-924f-edf5fb0ea2a7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7eb1a3a92ce841f4a6955bc13a3bc6e0', 'request_id': 'req-693acbf5-4dc3-40cc-924f-edf5fb0ea2a7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:48.889758 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-693acbf5-4dc3-40cc-924f-edf5fb0ea2a7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7eb1a3a92ce841f4a6955bc13a3bc6e0', 'request_id': 'req-693acbf5-4dc3-40cc-924f-edf5fb0ea2a7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:48.904442 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-693acbf5-4dc3-40cc-924f-edf5fb0ea2a7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7eb1a3a92ce841f4a6955bc13a3bc6e0', 'request_id': 'req-693acbf5-4dc3-40cc-924f-edf5fb0ea2a7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:48.978259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-693acbf5-4dc3-40cc-924f-edf5fb0ea2a7 admin admin] The token cache is being invalidate because project 7eb1a3a92ce841f4a6955bc13a3bc6e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:05:48.980605 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-693acbf5-4dc3-40cc-924f-edf5fb0ea2a7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:48.981689 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-693acbf5-4dc3-40cc-924f-edf5fb0ea2a7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:05:48.984601 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 941/1869] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:05:48 2022] DELETE /identity/v3/projects/7eb1a3a92ce841f4a6955bc13a3bc6e0 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:48.996221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ebf5cf-4efa-4c40-a352-a27c2285c97d None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:48.996685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ebf5cf-4efa-4c40-a352-a27c2285c97d None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:48.996784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ebf5cf-4efa-4c40-a352-a27c2285c97d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:49.018488 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-73ebf5cf-4efa-4c40-a352-a27c2285c97d None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:49.019710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-73ebf5cf-4efa-4c40-a352-a27c2285c97d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:49.068943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:49.082581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 929/1870] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:05:49.091663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c495e70-cacb-4e5e-91d7-f2ce4c54fd13 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:49.092138 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c495e70-cacb-4e5e-91d7-f2ce4c54fd13 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:49.092422 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c495e70-cacb-4e5e-91d7-f2ce4c54fd13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:49.114034 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-1c495e70-cacb-4e5e-91d7-f2ce4c54fd13 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:49.115262 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-1c495e70-cacb-4e5e-91d7-f2ce4c54fd13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:49.123481 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 942/1871] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:49.127764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ebf5cf-4efa-4c40-a352-a27c2285c97d None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:49.138226 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ebf5cf-4efa-4c40-a352-a27c2285c97d None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:49.144048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e368fc-4eb9-43d6-ae4b-f88eed17a7ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:49.145387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e368fc-4eb9-43d6-ae4b-f88eed17a7ef admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:49.145669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e368fc-4eb9-43d6-ae4b-f88eed17a7ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:49.149503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e368fc-4eb9-43d6-ae4b-f88eed17a7ef admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:49.149503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e368fc-4eb9-43d6-ae4b-f88eed17a7ef admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:05:49.149503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e368fc-4eb9-43d6-ae4b-f88eed17a7ef admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:49.149503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e368fc-4eb9-43d6-ae4b-f88eed17a7ef admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:49.161482 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 930/1872] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:05:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:05:49.166035 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c495e70-cacb-4e5e-91d7-f2ce4c54fd13 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:49.176231 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c495e70-cacb-4e5e-91d7-f2ce4c54fd13 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:49.181132 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2c06fa-4bc7-41d0-a245-0910dd67e807 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:49.181840 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2c06fa-4bc7-41d0-a245-0910dd67e807 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:49.182127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2c06fa-4bc7-41d0-a245-0910dd67e807 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:49.182419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2c06fa-4bc7-41d0-a245-0910dd67e807 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:49.183061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2c06fa-4bc7-41d0-a245-0910dd67e807 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:49.184995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2c06fa-4bc7-41d0-a245-0910dd67e807 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:49.267459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 943/1873] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:49 2022] POST /identity/v3/projects => generated 375 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:49.272767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e368fc-4eb9-43d6-ae4b-f88eed17a7ef admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:49.284998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e368fc-4eb9-43d6-ae4b-f88eed17a7ef admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:49.356539 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834e8f3d-e7d2-4672-a01b-6af53fd8d33d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:49.357570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834e8f3d-e7d2-4672-a01b-6af53fd8d33d admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:49.357854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834e8f3d-e7d2-4672-a01b-6af53fd8d33d admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:49.358156 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834e8f3d-e7d2-4672-a01b-6af53fd8d33d admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:49.358827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834e8f3d-e7d2-4672-a01b-6af53fd8d33d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:49.361250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834e8f3d-e7d2-4672-a01b-6af53fd8d33d admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:49.461906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 931/1874] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:49 2022] POST /identity/v3/users => generated 288 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:49.466293 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2c06fa-4bc7-41d0-a245-0910dd67e807 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:49.479057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2c06fa-4bc7-41d0-a245-0910dd67e807 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:49.485145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af98541a-c91c-435d-81c8-c40a96360a5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:49.486337 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af98541a-c91c-435d-81c8-c40a96360a5b admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:49.487002 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af98541a-c91c-435d-81c8-c40a96360a5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:49.487515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af98541a-c91c-435d-81c8-c40a96360a5b admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:49.488351 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af98541a-c91c-435d-81c8-c40a96360a5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:49.491178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af98541a-c91c-435d-81c8-c40a96360a5b admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:49.511531 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 944/1875] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:49 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:05:49.515196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834e8f3d-e7d2-4672-a01b-6af53fd8d33d admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:49.527135 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834e8f3d-e7d2-4672-a01b-6af53fd8d33d admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:49.538384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f98ffb1-aae4-42ec-ad88-6b94a59702e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:49.541459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f98ffb1-aae4-42ec-ad88-6b94a59702e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:49.542095 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f98ffb1-aae4-42ec-ad88-6b94a59702e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:49.542524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f98ffb1-aae4-42ec-ad88-6b94a59702e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:49.544602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f98ffb1-aae4-42ec-ad88-6b94a59702e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:49.546971 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f98ffb1-aae4-42ec-ad88-6b94a59702e1 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:49.563798 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 932/1876] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:49 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:49.568454 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af98541a-c91c-435d-81c8-c40a96360a5b admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:49.580570 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af98541a-c91c-435d-81c8-c40a96360a5b admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:49.586027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f421d927-8bce-4377-9b15-2ec90d14dd1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:49.586758 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f421d927-8bce-4377-9b15-2ec90d14dd1d admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:49.587105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f421d927-8bce-4377-9b15-2ec90d14dd1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:49.587393 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f421d927-8bce-4377-9b15-2ec90d14dd1d admin admin] PATH_INFO: `/v3/projects/3d5b10aa7be846c89453abdba743b025/users/4e46e2b0d7d046c9b5fcbe4ee2bac2b5/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:49.604237 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f421d927-8bce-4377-9b15-2ec90d14dd1d admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d5b10aa7be846c89453abdba743b025, user_id=4e46e2b0d7d046c9b5fcbe4ee2bac2b5, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:49.606916 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f421d927-8bce-4377-9b15-2ec90d14dd1d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:49.677799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 945/1877] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:49 2022] PUT /identity/v3/projects/3d5b10aa7be846c89453abdba743b025/users/4e46e2b0d7d046c9b5fcbe4ee2bac2b5/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:49.688664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8019d733-e927-4172-ab38-f50b92de88f9 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:49.688966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8019d733-e927-4172-ab38-f50b92de88f9 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:49.689347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8019d733-e927-4172-ab38-f50b92de88f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:49.725798 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-8019d733-e927-4172-ab38-f50b92de88f9 None None] MFA Rules not processed for user `4e46e2b0d7d046c9b5fcbe4ee2bac2b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:49.727246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8019d733-e927-4172-ab38-f50b92de88f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:49.798973 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 933/1878] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:49 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:05:49.803519 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f421d927-8bce-4377-9b15-2ec90d14dd1d admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:49.866044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f421d927-8bce-4377-9b15-2ec90d14dd1d admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:49.870767 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bdd8e6-2dfc-41bd-a063-6f48cf54b8d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:49.871376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bdd8e6-2dfc-41bd-a063-6f48cf54b8d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:49.871659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bdd8e6-2dfc-41bd-a063-6f48cf54b8d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:49.871933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93bdd8e6-2dfc-41bd-a063-6f48cf54b8d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:49.872576 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93bdd8e6-2dfc-41bd-a063-6f48cf54b8d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:49.874577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93bdd8e6-2dfc-41bd-a063-6f48cf54b8d9 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:50.009092 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 946/1879] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:49 2022] POST /identity/v3/projects => generated 375 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:50.010149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8019d733-e927-4172-ab38-f50b92de88f9 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:50.025509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8019d733-e927-4172-ab38-f50b92de88f9 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:50.083276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:50.097027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b778b85-58d8-444d-81c5-7c5aa63cefb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:50.097677 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b778b85-58d8-444d-81c5-7c5aa63cefb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:50.097960 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b778b85-58d8-444d-81c5-7c5aa63cefb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:50.098236 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b778b85-58d8-444d-81c5-7c5aa63cefb0 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:50.098891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b778b85-58d8-444d-81c5-7c5aa63cefb0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:50.100856 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b778b85-58d8-444d-81c5-7c5aa63cefb0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:50.214667 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 934/1880] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:50 2022] POST /identity/v3/users => generated 288 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:50.218419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bdd8e6-2dfc-41bd-a063-6f48cf54b8d9 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:50.230019 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93bdd8e6-2dfc-41bd-a063-6f48cf54b8d9 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:50.234493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a37dd4-1c20-4855-aadc-df890092ef7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:50.235090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a37dd4-1c20-4855-aadc-df890092ef7d admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:50.235397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a37dd4-1c20-4855-aadc-df890092ef7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:50.235665 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a37dd4-1c20-4855-aadc-df890092ef7d admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:50.236773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a37dd4-1c20-4855-aadc-df890092ef7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:50.238433 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a37dd4-1c20-4855-aadc-df890092ef7d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:50.248857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 947/1881] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:50 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:50.253036 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b778b85-58d8-444d-81c5-7c5aa63cefb0 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:50.262892 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b778b85-58d8-444d-81c5-7c5aa63cefb0 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:50.267452 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7e39fa-646f-48aa-b52e-1d1ce7c57bb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:50.268546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7e39fa-646f-48aa-b52e-1d1ce7c57bb4 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:50.268822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7e39fa-646f-48aa-b52e-1d1ce7c57bb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:50.269123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7e39fa-646f-48aa-b52e-1d1ce7c57bb4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:50.269782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7e39fa-646f-48aa-b52e-1d1ce7c57bb4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:50.271394 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7e39fa-646f-48aa-b52e-1d1ce7c57bb4 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:50.281289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 935/1882] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:50 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:50.285833 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a37dd4-1c20-4855-aadc-df890092ef7d admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:50.290989 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:50.296133 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a37dd4-1c20-4855-aadc-df890092ef7d admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:50.300786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc7a552-d912-4f4a-bbfa-069c1c2d0ed7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:50.301472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc7a552-d912-4f4a-bbfa-069c1c2d0ed7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:50.301754 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc7a552-d912-4f4a-bbfa-069c1c2d0ed7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:50.302028 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc7a552-d912-4f4a-bbfa-069c1c2d0ed7 admin admin] PATH_INFO: `/v3/projects/465c8420550d4f75a178d3ccfbf4d501/users/1651c2cdb81c49d387ceed8bf382c091/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:50.317399 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc7a552-d912-4f4a-bbfa-069c1c2d0ed7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=465c8420550d4f75a178d3ccfbf4d501, user_id=1651c2cdb81c49d387ceed8bf382c091, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:50.320035 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc7a552-d912-4f4a-bbfa-069c1c2d0ed7 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:50.379605 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 948/1883] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:50 2022] PUT /identity/v3/projects/465c8420550d4f75a178d3ccfbf4d501/users/1651c2cdb81c49d387ceed8bf382c091/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:50.390791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ad94bc-b3a5-464a-ba94-8d19e73e9086 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:50.391074 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ad94bc-b3a5-464a-ba94-8d19e73e9086 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:50.391350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ad94bc-b3a5-464a-ba94-8d19e73e9086 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:50.420580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-82ad94bc-b3a5-464a-ba94-8d19e73e9086 None None] MFA Rules not processed for user `1651c2cdb81c49d387ceed8bf382c091`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:50.421856 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-82ad94bc-b3a5-464a-ba94-8d19e73e9086 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:50.486943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 936/1884] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:50 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:50.491585 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc7a552-d912-4f4a-bbfa-069c1c2d0ed7 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:50.551538 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc7a552-d912-4f4a-bbfa-069c1c2d0ed7 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:50.556044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ea97b8-c564-4f51-b61b-55f13503d8be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:50.556637 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ea97b8-c564-4f51-b61b-55f13503d8be admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:50.556912 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ea97b8-c564-4f51-b61b-55f13503d8be admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:50.557281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ea97b8-c564-4f51-b61b-55f13503d8be admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:50.557900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ea97b8-c564-4f51-b61b-55f13503d8be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:50.559779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ea97b8-c564-4f51-b61b-55f13503d8be admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:50.645230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 949/1885] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:05:50 2022] POST /identity/v3/projects => generated 377 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:50.649725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ad94bc-b3a5-464a-ba94-8d19e73e9086 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:50.660551 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82ad94bc-b3a5-464a-ba94-8d19e73e9086 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:50.726740 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459a2b0a-7e00-40fa-89cd-4804e7eb6a67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:50.727189 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459a2b0a-7e00-40fa-89cd-4804e7eb6a67 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:50.727466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459a2b0a-7e00-40fa-89cd-4804e7eb6a67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:50.727734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459a2b0a-7e00-40fa-89cd-4804e7eb6a67 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:50.728366 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459a2b0a-7e00-40fa-89cd-4804e7eb6a67 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:50.730285 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459a2b0a-7e00-40fa-89cd-4804e7eb6a67 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:50.806871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 937/1886] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:05:50 2022] POST /identity/v3/users => generated 295 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:05:50.810973 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ea97b8-c564-4f51-b61b-55f13503d8be admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:50.821896 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ea97b8-c564-4f51-b61b-55f13503d8be admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:50.826549 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f16e18e-7b24-44d9-baed-c19df259f94c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:50.827143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f16e18e-7b24-44d9-baed-c19df259f94c admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:50.827423 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f16e18e-7b24-44d9-baed-c19df259f94c admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:50.827696 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f16e18e-7b24-44d9-baed-c19df259f94c admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:50.828328 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f16e18e-7b24-44d9-baed-c19df259f94c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:50.829970 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f16e18e-7b24-44d9-baed-c19df259f94c admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:50.840752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 950/1887] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:50 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:50.844988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459a2b0a-7e00-40fa-89cd-4804e7eb6a67 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:50.855211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459a2b0a-7e00-40fa-89cd-4804e7eb6a67 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:50.859927 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab0d6f4-5dc9-419a-b785-1a16df759e7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:50.860564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab0d6f4-5dc9-419a-b785-1a16df759e7c admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:50.861138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab0d6f4-5dc9-419a-b785-1a16df759e7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:50.861984 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab0d6f4-5dc9-419a-b785-1a16df759e7c admin admin] PATH_INFO: `/v3/projects/4612d1d746474c4ea901ab702b744f99/users/ce9ce2d0d06f4b059d05d68f588c9e66/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:50.876403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab0d6f4-5dc9-419a-b785-1a16df759e7c admin admin] RBAC: Authorizing `identity:create_grant(project_id=4612d1d746474c4ea901ab702b744f99, user_id=ce9ce2d0d06f4b059d05d68f588c9e66, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:50.879722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab0d6f4-5dc9-419a-b785-1a16df759e7c admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:50.960437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 938/1888] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:50 2022] PUT /identity/v3/projects/4612d1d746474c4ea901ab702b744f99/users/ce9ce2d0d06f4b059d05d68f588c9e66/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:50.970065 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f16e18e-7b24-44d9-baed-c19df259f94c admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:50.980084 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f16e18e-7b24-44d9-baed-c19df259f94c admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:51.037307 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb428ed-23cd-49d4-8316-61b9ac65b2b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:51.037988 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb428ed-23cd-49d4-8316-61b9ac65b2b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:51.038175 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb428ed-23cd-49d4-8316-61b9ac65b2b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:51.038447 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb428ed-23cd-49d4-8316-61b9ac65b2b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:51.039077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb428ed-23cd-49d4-8316-61b9ac65b2b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:51.040669 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb428ed-23cd-49d4-8316-61b9ac65b2b1 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:51.050884 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 951/1889] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:05:50 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:05:51.054924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab0d6f4-5dc9-419a-b785-1a16df759e7c admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:51.065799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab0d6f4-5dc9-419a-b785-1a16df759e7c admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:51.070615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2402b26e-cf8f-4f02-a9e6-538fde0e0608 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:51.071292 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2402b26e-cf8f-4f02-a9e6-538fde0e0608 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:51.071571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2402b26e-cf8f-4f02-a9e6-538fde0e0608 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:51.071841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2402b26e-cf8f-4f02-a9e6-538fde0e0608 admin admin] PATH_INFO: `/v3/projects/4612d1d746474c4ea901ab702b744f99/users/ce9ce2d0d06f4b059d05d68f588c9e66/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:51.073790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2402b26e-cf8f-4f02-a9e6-538fde0e0608 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4612d1d746474c4ea901ab702b744f99, user_id=ce9ce2d0d06f4b059d05d68f588c9e66, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:51.076439 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2402b26e-cf8f-4f02-a9e6-538fde0e0608 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:51.099772 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 939/1890] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:05:51 2022] PUT /identity/v3/projects/4612d1d746474c4ea901ab702b744f99/users/ce9ce2d0d06f4b059d05d68f588c9e66/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:05:51.110095 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65001e6-0619-4204-96d6-77efb92d7860 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:51.110573 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65001e6-0619-4204-96d6-77efb92d7860 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:51.111888 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65001e6-0619-4204-96d6-77efb92d7860 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:51.140853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-a65001e6-0619-4204-96d6-77efb92d7860 None None] MFA Rules not processed for user `ce9ce2d0d06f4b059d05d68f588c9e66`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:05:51.142102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a65001e6-0619-4204-96d6-77efb92d7860 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:51.188146 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:51.224885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 952/1891] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:05:51 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:05:51.237445 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2402b26e-cf8f-4f02-a9e6-538fde0e0608 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:51.239749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2402b26e-cf8f-4f02-a9e6-538fde0e0608 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:51.253239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:05:51.320657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2402b26e-cf8f-4f02-a9e6-538fde0e0608 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:51.325410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc1f63c-65ad-432a-a7a5-92746582d65c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:51.326396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc1f63c-65ad-432a-a7a5-92746582d65c service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:51.326960 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc1f63c-65ad-432a-a7a5-92746582d65c service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:51.327382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc1f63c-65ad-432a-a7a5-92746582d65c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:51.401222 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc1f63c-65ad-432a-a7a5-92746582d65c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:51.402722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc1f63c-65ad-432a-a7a5-92746582d65c service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:51.414019 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 940/1892] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:51 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:52.559044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65001e6-0619-4204-96d6-77efb92d7860 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:52.560589 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a65001e6-0619-4204-96d6-77efb92d7860 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:52.578267 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65001e6-0619-4204-96d6-77efb92d7860 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:52.582902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f284762-28bd-4908-860f-0e1ee67a1c5c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:52.583860 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f284762-28bd-4908-860f-0e1ee67a1c5c service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:52.584217 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f284762-28bd-4908-860f-0e1ee67a1c5c service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:52.584566 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f284762-28bd-4908-860f-0e1ee67a1c5c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:52.587089 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0f284762-28bd-4908-860f-0e1ee67a1c5c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:05:52.676685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f284762-28bd-4908-860f-0e1ee67a1c5c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:52.680216 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f284762-28bd-4908-860f-0e1ee67a1c5c service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:52.695711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 953/1893] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:52 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:54.268692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc1f63c-65ad-432a-a7a5-92746582d65c service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:54.277971 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc1f63c-65ad-432a-a7a5-92746582d65c service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:54.290591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d117c17-5a0f-41ef-a28c-2122c3c389bf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:54.291240 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d117c17-5a0f-41ef-a28c-2122c3c389bf service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:54.291515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d117c17-5a0f-41ef-a28c-2122c3c389bf service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:54.291786 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d117c17-5a0f-41ef-a28c-2122c3c389bf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:54.361571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d117c17-5a0f-41ef-a28c-2122c3c389bf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:54.363644 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d117c17-5a0f-41ef-a28c-2122c3c389bf service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:54.374213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 941/1894] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:54 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:05:58.274467 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f284762-28bd-4908-860f-0e1ee67a1c5c service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:05:58.283222 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f284762-28bd-4908-860f-0e1ee67a1c5c service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:05:58.287225 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759364b3-9f42-4154-bb82-40801198a370 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:05:58.288725 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759364b3-9f42-4154-bb82-40801198a370 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:05:58.288725 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759364b3-9f42-4154-bb82-40801198a370 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:05:58.288725 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759364b3-9f42-4154-bb82-40801198a370 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:05:58.297794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759364b3-9f42-4154-bb82-40801198a370 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:05:58.299214 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759364b3-9f42-4154-bb82-40801198a370 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:05:58.310113 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 954/1895] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:05:58 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:00.894285 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d117c17-5a0f-41ef-a28c-2122c3c389bf service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:00.901018 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2d117c17-5a0f-41ef-a28c-2122c3c389bf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:00.978643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d117c17-5a0f-41ef-a28c-2122c3c389bf service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:00.983281 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84868d48-8eee-44f5-9061-c5ce7d41d0c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:00.984018 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84868d48-8eee-44f5-9061-c5ce7d41d0c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:00.984295 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84868d48-8eee-44f5-9061-c5ce7d41d0c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:00.984563 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84868d48-8eee-44f5-9061-c5ce7d41d0c9 admin admin] PATH_INFO: `/v3/users/ee20b7bc96e940a0b6fd7dd8284270e9` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:00.986001 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84868d48-8eee-44f5-9061-c5ce7d41d0c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee20b7bc96e940a0b6fd7dd8284270e9)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:00.988522 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84868d48-8eee-44f5-9061-c5ce7d41d0c9 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:00.990255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:01.164730 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-84868d48-8eee-44f5-9061-c5ce7d41d0c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee20b7bc96e940a0b6fd7dd8284270e9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:01.165333 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-84868d48-8eee-44f5-9061-c5ce7d41d0c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee20b7bc96e940a0b6fd7dd8284270e9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:01.275952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-84868d48-8eee-44f5-9061-c5ce7d41d0c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee20b7bc96e940a0b6fd7dd8284270e9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:01.291603 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-84868d48-8eee-44f5-9061-c5ce7d41d0c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee20b7bc96e940a0b6fd7dd8284270e9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:01.291603 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-84868d48-8eee-44f5-9061-c5ce7d41d0c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee20b7bc96e940a0b6fd7dd8284270e9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:01.304101 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 942/1896] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:00 2022] DELETE /identity/v3/users/ee20b7bc96e940a0b6fd7dd8284270e9 => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:01.308873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a5cc7a-645c-4b65-9b99-d1665d0cd367 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:01.309518 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a5cc7a-645c-4b65-9b99-d1665d0cd367 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:01.309518 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a5cc7a-645c-4b65-9b99-d1665d0cd367 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:01.338957 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-49a5cc7a-645c-4b65-9b99-d1665d0cd367 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:01.340253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-49a5cc7a-645c-4b65-9b99-d1665d0cd367 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:01.426205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 955/1897] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:06:01.433660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84868d48-8eee-44f5-9061-c5ce7d41d0c9 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:01.435120 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-84868d48-8eee-44f5-9061-c5ce7d41d0c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:01.512978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84868d48-8eee-44f5-9061-c5ce7d41d0c9 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:01.517617 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401bbb7c-92e0-47ca-8740-5541f02967d5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:01.518241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401bbb7c-92e0-47ca-8740-5541f02967d5 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:01.519159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401bbb7c-92e0-47ca-8740-5541f02967d5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:01.519383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401bbb7c-92e0-47ca-8740-5541f02967d5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:01.530820 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401bbb7c-92e0-47ca-8740-5541f02967d5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:01.532831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401bbb7c-92e0-47ca-8740-5541f02967d5 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:01.545207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 943/1898] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:02.165862 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a5cc7a-645c-4b65-9b99-d1665d0cd367 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:02.169005 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-49a5cc7a-645c-4b65-9b99-d1665d0cd367 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:02.190994 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a5cc7a-645c-4b65-9b99-d1665d0cd367 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:02.195861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e26c1b4-a7de-428b-9c27-3451342fb593 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:02.196511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e26c1b4-a7de-428b-9c27-3451342fb593 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:02.196869 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e26c1b4-a7de-428b-9c27-3451342fb593 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:02.197041 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e26c1b4-a7de-428b-9c27-3451342fb593 admin admin] PATH_INFO: `/v3/projects/8d0a8304bc0845249f776ea53f3567cf` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:02.198687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e26c1b4-a7de-428b-9c27-3451342fb593 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d0a8304bc0845249f776ea53f3567cf)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:02.200731 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e26c1b4-a7de-428b-9c27-3451342fb593 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:02.345595 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-9e26c1b4-a7de-428b-9c27-3451342fb593 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d0a8304bc0845249f776ea53f3567cf', 'request_id': 'req-9e26c1b4-a7de-428b-9c27-3451342fb593'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:02.349712 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:02.351815 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-9e26c1b4-a7de-428b-9c27-3451342fb593 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d0a8304bc0845249f776ea53f3567cf', 'request_id': 'req-9e26c1b4-a7de-428b-9c27-3451342fb593'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:02.358321 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-9e26c1b4-a7de-428b-9c27-3451342fb593 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d0a8304bc0845249f776ea53f3567cf', 'request_id': 'req-9e26c1b4-a7de-428b-9c27-3451342fb593'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:02.470525 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-9e26c1b4-a7de-428b-9c27-3451342fb593 admin admin] The token cache is being invalidate because project 8d0a8304bc0845249f776ea53f3567cf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:02.470758 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-9e26c1b4-a7de-428b-9c27-3451342fb593 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:02.471399 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-9e26c1b4-a7de-428b-9c27-3451342fb593 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:02.473100 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 956/1899] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:02 2022] DELETE /identity/v3/projects/8d0a8304bc0845249f776ea53f3567cf => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:02.477159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401bbb7c-92e0-47ca-8740-5541f02967d5 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:02.487224 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401bbb7c-92e0-47ca-8740-5541f02967d5 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:02.544973 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b7ecca-4f50-4698-aed1-c6203ca3325c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:02.545746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b7ecca-4f50-4698-aed1-c6203ca3325c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:02.546102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b7ecca-4f50-4698-aed1-c6203ca3325c admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:02.546451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b7ecca-4f50-4698-aed1-c6203ca3325c admin admin] PATH_INFO: `/v3/users/d92b5fa9dc9d4358beac4808a070d9be` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:02.547654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b7ecca-4f50-4698-aed1-c6203ca3325c admin admin] RBAC: Authorizing `identity:delete_user(user_id=d92b5fa9dc9d4358beac4808a070d9be)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:02.549695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b7ecca-4f50-4698-aed1-c6203ca3325c admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:02.575608 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:02.773925 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-59b7ecca-4f50-4698-aed1-c6203ca3325c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd92b5fa9dc9d4358beac4808a070d9be'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:02.774588 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-59b7ecca-4f50-4698-aed1-c6203ca3325c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd92b5fa9dc9d4358beac4808a070d9be'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:02.835872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-59b7ecca-4f50-4698-aed1-c6203ca3325c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd92b5fa9dc9d4358beac4808a070d9be'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:02.849205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-59b7ecca-4f50-4698-aed1-c6203ca3325c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd92b5fa9dc9d4358beac4808a070d9be'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:02.849885 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-59b7ecca-4f50-4698-aed1-c6203ca3325c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd92b5fa9dc9d4358beac4808a070d9be'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:02.862710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 944/1900] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:02 2022] DELETE /identity/v3/users/d92b5fa9dc9d4358beac4808a070d9be => generated 0 bytes in 386 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:06:03.308629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e26c1b4-a7de-428b-9c27-3451342fb593 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:03.310680 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9e26c1b4-a7de-428b-9c27-3451342fb593 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:03.325348 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e26c1b4-a7de-428b-9c27-3451342fb593 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:03.388699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0461ac11-316e-4fc7-b005-d74d7ee044ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:03.389387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0461ac11-316e-4fc7-b005-d74d7ee044ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:03.389666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0461ac11-316e-4fc7-b005-d74d7ee044ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:03.389941 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0461ac11-316e-4fc7-b005-d74d7ee044ee admin admin] PATH_INFO: `/v3/projects/174be28425554302a7e0829820fda15b` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:03.391084 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0461ac11-316e-4fc7-b005-d74d7ee044ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=174be28425554302a7e0829820fda15b)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:03.392988 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0461ac11-316e-4fc7-b005-d74d7ee044ee admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:03.539878 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0461ac11-316e-4fc7-b005-d74d7ee044ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '174be28425554302a7e0829820fda15b', 'request_id': 'req-0461ac11-316e-4fc7-b005-d74d7ee044ee'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:03.545768 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0461ac11-316e-4fc7-b005-d74d7ee044ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '174be28425554302a7e0829820fda15b', 'request_id': 'req-0461ac11-316e-4fc7-b005-d74d7ee044ee'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:03.552527 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0461ac11-316e-4fc7-b005-d74d7ee044ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '174be28425554302a7e0829820fda15b', 'request_id': 'req-0461ac11-316e-4fc7-b005-d74d7ee044ee'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:03.635991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0461ac11-316e-4fc7-b005-d74d7ee044ee admin admin] The token cache is being invalidate because project 174be28425554302a7e0829820fda15b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:03.636285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0461ac11-316e-4fc7-b005-d74d7ee044ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:03.637004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0461ac11-316e-4fc7-b005-d74d7ee044ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:03.638592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 957/1901] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:03 2022] DELETE /identity/v3/projects/174be28425554302a7e0829820fda15b => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:03.642719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b7ecca-4f50-4698-aed1-c6203ca3325c admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:03.644768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-59b7ecca-4f50-4698-aed1-c6203ca3325c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:03.658806 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b7ecca-4f50-4698-aed1-c6203ca3325c admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:03.724258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad74fb30-2c4d-46cf-9273-eaef0a69a5e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:03.725004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad74fb30-2c4d-46cf-9273-eaef0a69a5e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:03.725488 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad74fb30-2c4d-46cf-9273-eaef0a69a5e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:03.725914 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad74fb30-2c4d-46cf-9273-eaef0a69a5e8 admin admin] PATH_INFO: `/v3/users/18f656bdeb014871b71b138280616beb` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:03.727264 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad74fb30-2c4d-46cf-9273-eaef0a69a5e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=18f656bdeb014871b71b138280616beb)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:03.730792 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad74fb30-2c4d-46cf-9273-eaef0a69a5e8 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:03.896535 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ad74fb30-2c4d-46cf-9273-eaef0a69a5e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '18f656bdeb014871b71b138280616beb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:03.897332 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ad74fb30-2c4d-46cf-9273-eaef0a69a5e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '18f656bdeb014871b71b138280616beb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:03.938852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ad74fb30-2c4d-46cf-9273-eaef0a69a5e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '18f656bdeb014871b71b138280616beb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:03.957750 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ad74fb30-2c4d-46cf-9273-eaef0a69a5e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '18f656bdeb014871b71b138280616beb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:03.962028 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ad74fb30-2c4d-46cf-9273-eaef0a69a5e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '18f656bdeb014871b71b138280616beb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:03.979469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 945/1902] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:03 2022] DELETE /identity/v3/users/18f656bdeb014871b71b138280616beb => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:04.437483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0461ac11-316e-4fc7-b005-d74d7ee044ee admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:04.438990 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0461ac11-316e-4fc7-b005-d74d7ee044ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:04.456251 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0461ac11-316e-4fc7-b005-d74d7ee044ee admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:04.526840 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82159815-cab5-4648-a1b4-8aabf1bde95d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:04.527488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82159815-cab5-4648-a1b4-8aabf1bde95d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:04.527789 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82159815-cab5-4648-a1b4-8aabf1bde95d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:04.528019 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82159815-cab5-4648-a1b4-8aabf1bde95d admin admin] PATH_INFO: `/v3/projects/760186835723424d857dde933c062bea` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:04.529183 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82159815-cab5-4648-a1b4-8aabf1bde95d admin admin] RBAC: Authorizing `identity:delete_project(project_id=760186835723424d857dde933c062bea)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:04.531101 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82159815-cab5-4648-a1b4-8aabf1bde95d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:04.547913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:DELETE=9,keystone:SELECT=89,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:04.655402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-82159815-cab5-4648-a1b4-8aabf1bde95d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '760186835723424d857dde933c062bea', 'request_id': 'req-82159815-cab5-4648-a1b4-8aabf1bde95d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:04.661032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-82159815-cab5-4648-a1b4-8aabf1bde95d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '760186835723424d857dde933c062bea', 'request_id': 'req-82159815-cab5-4648-a1b4-8aabf1bde95d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:04.667509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-82159815-cab5-4648-a1b4-8aabf1bde95d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '760186835723424d857dde933c062bea', 'request_id': 'req-82159815-cab5-4648-a1b4-8aabf1bde95d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:04.767790 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-82159815-cab5-4648-a1b4-8aabf1bde95d admin admin] The token cache is being invalidate because project 760186835723424d857dde933c062bea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:04.768081 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-82159815-cab5-4648-a1b4-8aabf1bde95d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:04.768717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-82159815-cab5-4648-a1b4-8aabf1bde95d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:04.770251 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 958/1903] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:04 2022] DELETE /identity/v3/projects/760186835723424d857dde933c062bea => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:04.776334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16ce006-5cd6-4853-90f9-8412bad74c59 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:04.776814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16ce006-5cd6-4853-90f9-8412bad74c59 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:04.777004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16ce006-5cd6-4853-90f9-8412bad74c59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:04.797923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-c16ce006-5cd6-4853-90f9-8412bad74c59 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:04.799142 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-c16ce006-5cd6-4853-90f9-8412bad74c59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:04.815685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:04.867497 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 946/1904] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:04.877109 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07c0248-a668-430b-becd-17092a991d5a None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:04.877394 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07c0248-a668-430b-becd-17092a991d5a None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:04.877671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07c0248-a668-430b-becd-17092a991d5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:04.897465 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-c07c0248-a668-430b-becd-17092a991d5a None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:04.898687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c07c0248-a668-430b-becd-17092a991d5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:04.909159 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 959/1905] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:04.913844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16ce006-5cd6-4853-90f9-8412bad74c59 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:04.925209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16ce006-5cd6-4853-90f9-8412bad74c59 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:04.934660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc49d7d4-e66c-41b3-b905-fa3c4a3bb17c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:04.935523 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc49d7d4-e66c-41b3-b905-fa3c4a3bb17c admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:04.937494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc49d7d4-e66c-41b3-b905-fa3c4a3bb17c admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:04.937942 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc49d7d4-e66c-41b3-b905-fa3c4a3bb17c admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:04.939601 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc49d7d4-e66c-41b3-b905-fa3c4a3bb17c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:06:04.940382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc49d7d4-e66c-41b3-b905-fa3c4a3bb17c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:04.942666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc49d7d4-e66c-41b3-b905-fa3c4a3bb17c admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:04.960489 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 947/1906] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:06:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:06:04.961781 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07c0248-a668-430b-becd-17092a991d5a None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:04.971253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07c0248-a668-430b-becd-17092a991d5a None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:04.976700 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acd7f18-b98e-432b-96ac-5fc1ff2d7f2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:04.977719 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acd7f18-b98e-432b-96ac-5fc1ff2d7f2d admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:04.977719 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acd7f18-b98e-432b-96ac-5fc1ff2d7f2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:04.977928 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acd7f18-b98e-432b-96ac-5fc1ff2d7f2d admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:04.978555 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acd7f18-b98e-432b-96ac-5fc1ff2d7f2d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:04.980440 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acd7f18-b98e-432b-96ac-5fc1ff2d7f2d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:05.065932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 960/1907] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:06:04 2022] POST /identity/v3/projects => generated 385 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:06:05.070558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc49d7d4-e66c-41b3-b905-fa3c4a3bb17c admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:05.080528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc49d7d4-e66c-41b3-b905-fa3c4a3bb17c admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:05.138918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a78c3d2-8ebb-4d22-912e-20b7b9cd60c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:05.139830 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a78c3d2-8ebb-4d22-912e-20b7b9cd60c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:05.140141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a78c3d2-8ebb-4d22-912e-20b7b9cd60c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:05.140413 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a78c3d2-8ebb-4d22-912e-20b7b9cd60c6 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:05.141038 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a78c3d2-8ebb-4d22-912e-20b7b9cd60c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:05.143021 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a78c3d2-8ebb-4d22-912e-20b7b9cd60c6 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:05.279642 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 948/1908] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:06:05 2022] POST /identity/v3/users => generated 293 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:06:05.281105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acd7f18-b98e-432b-96ac-5fc1ff2d7f2d admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:05.291760 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acd7f18-b98e-432b-96ac-5fc1ff2d7f2d admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:05.296221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285f3d69-99e5-4979-beb6-983d471ba366 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:05.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285f3d69-99e5-4979-beb6-983d471ba366 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:05.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285f3d69-99e5-4979-beb6-983d471ba366 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:05.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285f3d69-99e5-4979-beb6-983d471ba366 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:05.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285f3d69-99e5-4979-beb6-983d471ba366 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:05.302350 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285f3d69-99e5-4979-beb6-983d471ba366 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:05.310628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 961/1909] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:05 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:06:05.314586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a78c3d2-8ebb-4d22-912e-20b7b9cd60c6 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:05.325185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a78c3d2-8ebb-4d22-912e-20b7b9cd60c6 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:05.329867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e66963-54e2-4267-8ef2-616ac082637e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:05.330493 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e66963-54e2-4267-8ef2-616ac082637e admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:05.330769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e66963-54e2-4267-8ef2-616ac082637e admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:05.331061 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e66963-54e2-4267-8ef2-616ac082637e admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:05.332300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e66963-54e2-4267-8ef2-616ac082637e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:05.333926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e66963-54e2-4267-8ef2-616ac082637e admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:05.344673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 949/1910] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:06:05.349017 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285f3d69-99e5-4979-beb6-983d471ba366 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:05.359029 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285f3d69-99e5-4979-beb6-983d471ba366 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:05.365193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2609d259-0773-4f5a-a516-d2fd6811a1ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:05.365894 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2609d259-0773-4f5a-a516-d2fd6811a1ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:05.366262 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2609d259-0773-4f5a-a516-d2fd6811a1ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:05.366660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2609d259-0773-4f5a-a516-d2fd6811a1ff admin admin] PATH_INFO: `/v3/projects/ca0931d51497476f9f229632aa8e227b/users/e83ee3a37413487c87a16f4a3eaa92f0/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:05.384357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2609d259-0773-4f5a-a516-d2fd6811a1ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=ca0931d51497476f9f229632aa8e227b, user_id=e83ee3a37413487c87a16f4a3eaa92f0, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:05.388370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2609d259-0773-4f5a-a516-d2fd6811a1ff admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:05.480616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 962/1911] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:06:05 2022] PUT /identity/v3/projects/ca0931d51497476f9f229632aa8e227b/users/e83ee3a37413487c87a16f4a3eaa92f0/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:05.492830 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95ef66e-4c68-487c-a03a-84f56f19bc0a None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:05.493145 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95ef66e-4c68-487c-a03a-84f56f19bc0a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:05.493423 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95ef66e-4c68-487c-a03a-84f56f19bc0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:05.522860 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-d95ef66e-4c68-487c-a03a-84f56f19bc0a None None] MFA Rules not processed for user `e83ee3a37413487c87a16f4a3eaa92f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:05.524078 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d95ef66e-4c68-487c-a03a-84f56f19bc0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:05.589030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 950/1912] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:05 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:05.593536 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2609d259-0773-4f5a-a516-d2fd6811a1ff admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:05.672997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2609d259-0773-4f5a-a516-d2fd6811a1ff admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:05.679237 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2080dff-89d7-46a7-ab7d-9767834dc617 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:05.680056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2080dff-89d7-46a7-ab7d-9767834dc617 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:05.680609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2080dff-89d7-46a7-ab7d-9767834dc617 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:05.680991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2080dff-89d7-46a7-ab7d-9767834dc617 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:05.683878 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2080dff-89d7-46a7-ab7d-9767834dc617 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:05.684559 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2080dff-89d7-46a7-ab7d-9767834dc617 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:05.834929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 963/1913] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:06:05 2022] POST /identity/v3/projects => generated 385 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:06:05.840101 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95ef66e-4c68-487c-a03a-84f56f19bc0a None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:05.854339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95ef66e-4c68-487c-a03a-84f56f19bc0a None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:05.887160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:05.929819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2080dff-89d7-46a7-ab7d-9767834dc617 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:05.932546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b2080dff-89d7-46a7-ab7d-9767834dc617 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:05.967609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:05.973915 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942e2f1c-0cc7-4352-834f-03bd495af81b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:05.974772 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942e2f1c-0cc7-4352-834f-03bd495af81b admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:05.975127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942e2f1c-0cc7-4352-834f-03bd495af81b admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:05.975471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942e2f1c-0cc7-4352-834f-03bd495af81b admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:05.976601 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942e2f1c-0cc7-4352-834f-03bd495af81b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:05.982065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942e2f1c-0cc7-4352-834f-03bd495af81b admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:06.021857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2080dff-89d7-46a7-ab7d-9767834dc617 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:06.026625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d07905-8440-4e22-977b-ebcb289d11d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:06.027255 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d07905-8440-4e22-977b-ebcb289d11d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:06.027535 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d07905-8440-4e22-977b-ebcb289d11d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:06.027809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d07905-8440-4e22-977b-ebcb289d11d0 admin admin] PATH_INFO: `/v3/users/4e46e2b0d7d046c9b5fcbe4ee2bac2b5` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:06.028939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d07905-8440-4e22-977b-ebcb289d11d0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4e46e2b0d7d046c9b5fcbe4ee2bac2b5)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:06.030888 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d07905-8440-4e22-977b-ebcb289d11d0 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:06.089823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 951/1914] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:06:05 2022] POST /identity/v3/users => generated 299 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:06:06.091067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942e2f1c-0cc7-4352-834f-03bd495af81b admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:06.101169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942e2f1c-0cc7-4352-834f-03bd495af81b admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:06.106022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a453f3cd-c9ce-4d1b-a9de-9e0b8a69e6af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:06.106732 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a453f3cd-c9ce-4d1b-a9de-9e0b8a69e6af admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:06.107113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a453f3cd-c9ce-4d1b-a9de-9e0b8a69e6af admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:06.107711 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a453f3cd-c9ce-4d1b-a9de-9e0b8a69e6af admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:06.108211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a453f3cd-c9ce-4d1b-a9de-9e0b8a69e6af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:06.110088 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a453f3cd-c9ce-4d1b-a9de-9e0b8a69e6af admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:06.120990 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 952/1915] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:06 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:06:06.125128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a453f3cd-c9ce-4d1b-a9de-9e0b8a69e6af admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:06.135738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a453f3cd-c9ce-4d1b-a9de-9e0b8a69e6af admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:06.140387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e313567c-9b08-4e05-9f84-8d87ba685193 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:06.141046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e313567c-9b08-4e05-9f84-8d87ba685193 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:06.141355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e313567c-9b08-4e05-9f84-8d87ba685193 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:06.141629 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e313567c-9b08-4e05-9f84-8d87ba685193 admin admin] PATH_INFO: `/v3/projects/1e621e0c178344bd86a1cd86734c629f/users/aab386fa32db456a97ad1a1966b8d24d/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:06.157475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e313567c-9b08-4e05-9f84-8d87ba685193 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e621e0c178344bd86a1cd86734c629f, user_id=aab386fa32db456a97ad1a1966b8d24d, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:06.160164 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e313567c-9b08-4e05-9f84-8d87ba685193 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:06.247959 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 953/1916] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:06:06 2022] PUT /identity/v3/projects/1e621e0c178344bd86a1cd86734c629f/users/aab386fa32db456a97ad1a1966b8d24d/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:06:06.251865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e313567c-9b08-4e05-9f84-8d87ba685193 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:06.286794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-77d07905-8440-4e22-977b-ebcb289d11d0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4e46e2b0d7d046c9b5fcbe4ee2bac2b5'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:06.321054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e313567c-9b08-4e05-9f84-8d87ba685193 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:06.325792 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0064a025-83a6-435d-9d6a-da62d5ea1934 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:06.326530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0064a025-83a6-435d-9d6a-da62d5ea1934 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:06.326808 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0064a025-83a6-435d-9d6a-da62d5ea1934 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:06.327080 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0064a025-83a6-435d-9d6a-da62d5ea1934 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:06.327731 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0064a025-83a6-435d-9d6a-da62d5ea1934 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:06.329347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0064a025-83a6-435d-9d6a-da62d5ea1934 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:06.344587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 954/1917] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:06 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:06:06.345909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0064a025-83a6-435d-9d6a-da62d5ea1934 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:06.355595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0064a025-83a6-435d-9d6a-da62d5ea1934 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:06.360174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4f7b45-85c1-420f-97c5-2a8489c02c15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:06.361111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4f7b45-85c1-420f-97c5-2a8489c02c15 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:06.361420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4f7b45-85c1-420f-97c5-2a8489c02c15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:06.365880 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4f7b45-85c1-420f-97c5-2a8489c02c15 admin admin] PATH_INFO: `/v3/projects/1e621e0c178344bd86a1cd86734c629f/users/aab386fa32db456a97ad1a1966b8d24d/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:06.366112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-77d07905-8440-4e22-977b-ebcb289d11d0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4e46e2b0d7d046c9b5fcbe4ee2bac2b5'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:06.366207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4f7b45-85c1-420f-97c5-2a8489c02c15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e621e0c178344bd86a1cd86734c629f, user_id=aab386fa32db456a97ad1a1966b8d24d, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:06.366817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4f7b45-85c1-420f-97c5-2a8489c02c15 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:06.377775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 955/1918] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:06:06 2022] PUT /identity/v3/projects/1e621e0c178344bd86a1cd86734c629f/users/aab386fa32db456a97ad1a1966b8d24d/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:06.377953 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-77d07905-8440-4e22-977b-ebcb289d11d0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4e46e2b0d7d046c9b5fcbe4ee2bac2b5'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:06.379363 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-77d07905-8440-4e22-977b-ebcb289d11d0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4e46e2b0d7d046c9b5fcbe4ee2bac2b5'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:06.387137 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-77d07905-8440-4e22-977b-ebcb289d11d0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4e46e2b0d7d046c9b5fcbe4ee2bac2b5'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:06.395240 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e655e8e-5b98-4206-947b-2ebffc0906a9 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:06.395533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e655e8e-5b98-4206-947b-2ebffc0906a9 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:06.395806 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e655e8e-5b98-4206-947b-2ebffc0906a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:06.397805 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 964/1919] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:05 2022] DELETE /identity/v3/users/4e46e2b0d7d046c9b5fcbe4ee2bac2b5 => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:06.400839 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5adc1f5-a0a9-44a4-8bd1-578b80b8e045 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:06.401155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5adc1f5-a0a9-44a4-8bd1-578b80b8e045 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:06.401436 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5adc1f5-a0a9-44a4-8bd1-578b80b8e045 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:06.420059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-e5adc1f5-a0a9-44a4-8bd1-578b80b8e045 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:06.421296 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e5adc1f5-a0a9-44a4-8bd1-578b80b8e045 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:06.426524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-8e655e8e-5b98-4206-947b-2ebffc0906a9 None None] MFA Rules not processed for user `aab386fa32db456a97ad1a1966b8d24d`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:06.428188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8e655e8e-5b98-4206-947b-2ebffc0906a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:06.480134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:06.486618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 965/1920] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:06:06.496841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5adc1f5-a0a9-44a4-8bd1-578b80b8e045 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:06.498223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e5adc1f5-a0a9-44a4-8bd1-578b80b8e045 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:06.508602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 956/1921] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:06 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:06.522555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e655e8e-5b98-4206-947b-2ebffc0906a9 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:06.523895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8e655e8e-5b98-4206-947b-2ebffc0906a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:06.546140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:06.557554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5adc1f5-a0a9-44a4-8bd1-578b80b8e045 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:06.561905 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6084f14-5893-4625-82a6-f91ed04deaae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:06.562510 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6084f14-5893-4625-82a6-f91ed04deaae service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:06.562786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6084f14-5893-4625-82a6-f91ed04deaae service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:06.563051 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6084f14-5893-4625-82a6-f91ed04deaae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:06.572985 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6084f14-5893-4625-82a6-f91ed04deaae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:06.574519 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6084f14-5893-4625-82a6-f91ed04deaae service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:06.582939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e655e8e-5b98-4206-947b-2ebffc0906a9 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:06.584217 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 966/1922] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:06:06.587589 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ee0d71-4301-4430-a5ec-8ace09c57a3d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:06.588198 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ee0d71-4301-4430-a5ec-8ace09c57a3d service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:06.588477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ee0d71-4301-4430-a5ec-8ace09c57a3d service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:06.588777 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ee0d71-4301-4430-a5ec-8ace09c57a3d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:06.598363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ee0d71-4301-4430-a5ec-8ace09c57a3d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:06.599785 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ee0d71-4301-4430-a5ec-8ace09c57a3d service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:06.610152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 957/1923] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:06 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:06:07.081151 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6084f14-5893-4625-82a6-f91ed04deaae service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:07.084011 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d6084f14-5893-4625-82a6-f91ed04deaae service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:07.104219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6084f14-5893-4625-82a6-f91ed04deaae service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:07.116003 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:07.117311 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:07.117311 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:07.117500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f admin admin] PATH_INFO: `/v3/projects/3d5b10aa7be846c89453abdba743b025` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:07.119052 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f admin admin] RBAC: Authorizing `identity:delete_project(project_id=3d5b10aa7be846c89453abdba743b025)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:07.121694 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:07.325974 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3d5b10aa7be846c89453abdba743b025', 'request_id': 'req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:07.360258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3d5b10aa7be846c89453abdba743b025', 'request_id': 'req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:07.377500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3d5b10aa7be846c89453abdba743b025', 'request_id': 'req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:07.439401 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f admin admin] The token cache is being invalidate because project 3d5b10aa7be846c89453abdba743b025 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:07.440591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:07.445892 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:07.448743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 967/1924] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:07 2022] DELETE /identity/v3/projects/3d5b10aa7be846c89453abdba743b025 => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:07.453443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ee0d71-4301-4430-a5ec-8ace09c57a3d service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:07.471851 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ee0d71-4301-4430-a5ec-8ace09c57a3d service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:07.547723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5e4f56-29c6-43ed-abae-3dd92326a754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:07.548400 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5e4f56-29c6-43ed-abae-3dd92326a754 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:07.548683 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5e4f56-29c6-43ed-abae-3dd92326a754 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:07.548957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5e4f56-29c6-43ed-abae-3dd92326a754 admin admin] PATH_INFO: `/v3/users/1651c2cdb81c49d387ceed8bf382c091` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:07.550998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5e4f56-29c6-43ed-abae-3dd92326a754 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1651c2cdb81c49d387ceed8bf382c091)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:07.553014 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5e4f56-29c6-43ed-abae-3dd92326a754 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:07.876584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-9d5e4f56-29c6-43ed-abae-3dd92326a754 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1651c2cdb81c49d387ceed8bf382c091'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:07.877565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-9d5e4f56-29c6-43ed-abae-3dd92326a754 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1651c2cdb81c49d387ceed8bf382c091'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:07.966996 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-9d5e4f56-29c6-43ed-abae-3dd92326a754 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1651c2cdb81c49d387ceed8bf382c091'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:08.008035 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-9d5e4f56-29c6-43ed-abae-3dd92326a754 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1651c2cdb81c49d387ceed8bf382c091'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:08.008811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-9d5e4f56-29c6-43ed-abae-3dd92326a754 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1651c2cdb81c49d387ceed8bf382c091'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:08.024166 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:08.032088 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 958/1925] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:07 2022] DELETE /identity/v3/users/1651c2cdb81c49d387ceed8bf382c091 => generated 0 bytes in 576 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:08.086105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:08.088710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:08.121798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9fcefe-891b-4892-a3e3-0f7655d3ee6f admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:08.243829 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64ff6a1-2a99-4e0d-9b1a-44b6e03d8799 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:08.244494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64ff6a1-2a99-4e0d-9b1a-44b6e03d8799 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:08.244779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64ff6a1-2a99-4e0d-9b1a-44b6e03d8799 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:08.245058 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64ff6a1-2a99-4e0d-9b1a-44b6e03d8799 admin admin] PATH_INFO: `/v3/users/e83ee3a37413487c87a16f4a3eaa92f0` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:08.246323 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64ff6a1-2a99-4e0d-9b1a-44b6e03d8799 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e83ee3a37413487c87a16f4a3eaa92f0)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:08.248392 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64ff6a1-2a99-4e0d-9b1a-44b6e03d8799 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:08.485211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e64ff6a1-2a99-4e0d-9b1a-44b6e03d8799 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e83ee3a37413487c87a16f4a3eaa92f0'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:08.584544 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e64ff6a1-2a99-4e0d-9b1a-44b6e03d8799 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e83ee3a37413487c87a16f4a3eaa92f0'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:08.610028 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e64ff6a1-2a99-4e0d-9b1a-44b6e03d8799 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e83ee3a37413487c87a16f4a3eaa92f0'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:08.612529 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e64ff6a1-2a99-4e0d-9b1a-44b6e03d8799 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e83ee3a37413487c87a16f4a3eaa92f0'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:08.618194 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:08.650564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e64ff6a1-2a99-4e0d-9b1a-44b6e03d8799 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e83ee3a37413487c87a16f4a3eaa92f0'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:08.671026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 968/1926] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:08 2022] DELETE /identity/v3/users/e83ee3a37413487c87a16f4a3eaa92f0 => generated 0 bytes in 585 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:08.678289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9666c4c5-02a8-4862-b391-f3a65218629f None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:08.678829 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9666c4c5-02a8-4862-b391-f3a65218629f None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:08.678918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9666c4c5-02a8-4862-b391-f3a65218629f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:08.723518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-9666c4c5-02a8-4862-b391-f3a65218629f None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:08.725422 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-9666c4c5-02a8-4862-b391-f3a65218629f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:08.807158 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 959/1927] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:08.837580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64ff6a1-2a99-4e0d-9b1a-44b6e03d8799 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:08.839324 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e64ff6a1-2a99-4e0d-9b1a-44b6e03d8799 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:08.943137 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64ff6a1-2a99-4e0d-9b1a-44b6e03d8799 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:08.948610 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4dd04f-aefa-4209-a798-2776c3a023f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:08.949366 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4dd04f-aefa-4209-a798-2776c3a023f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:08.949721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4dd04f-aefa-4209-a798-2776c3a023f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:08.950078 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4dd04f-aefa-4209-a798-2776c3a023f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:08.981746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4dd04f-aefa-4209-a798-2776c3a023f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:08.983272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4dd04f-aefa-4209-a798-2776c3a023f0 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:08.995022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 969/1928] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:09.062840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9666c4c5-02a8-4862-b391-f3a65218629f None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:09.064507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-9666c4c5-02a8-4862-b391-f3a65218629f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:09.077979 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9666c4c5-02a8-4862-b391-f3a65218629f None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:09.084318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8711a5af-27c0-486b-8e94-ea6e11daa603 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:09.088991 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8711a5af-27c0-486b-8e94-ea6e11daa603 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:09.088991 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8711a5af-27c0-486b-8e94-ea6e11daa603 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:09.088991 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8711a5af-27c0-486b-8e94-ea6e11daa603 admin admin] PATH_INFO: `/v3/projects/465c8420550d4f75a178d3ccfbf4d501` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:09.088991 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8711a5af-27c0-486b-8e94-ea6e11daa603 admin admin] RBAC: Authorizing `identity:delete_project(project_id=465c8420550d4f75a178d3ccfbf4d501)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:09.090617 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8711a5af-27c0-486b-8e94-ea6e11daa603 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:09.330388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8711a5af-27c0-486b-8e94-ea6e11daa603 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '465c8420550d4f75a178d3ccfbf4d501', 'request_id': 'req-8711a5af-27c0-486b-8e94-ea6e11daa603'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:09.337905 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8711a5af-27c0-486b-8e94-ea6e11daa603 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '465c8420550d4f75a178d3ccfbf4d501', 'request_id': 'req-8711a5af-27c0-486b-8e94-ea6e11daa603'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:09.407870 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8711a5af-27c0-486b-8e94-ea6e11daa603 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '465c8420550d4f75a178d3ccfbf4d501', 'request_id': 'req-8711a5af-27c0-486b-8e94-ea6e11daa603'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:09.421727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8711a5af-27c0-486b-8e94-ea6e11daa603 admin admin] The token cache is being invalidate because project 465c8420550d4f75a178d3ccfbf4d501 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:09.422068 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8711a5af-27c0-486b-8e94-ea6e11daa603 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:09.423097 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8711a5af-27c0-486b-8e94-ea6e11daa603 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:09.431880 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 960/1929] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:09 2022] DELETE /identity/v3/projects/465c8420550d4f75a178d3ccfbf4d501 => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:09.435546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4dd04f-aefa-4209-a798-2776c3a023f0 service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:09.446288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4dd04f-aefa-4209-a798-2776c3a023f0 service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:09.512037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940d8365-9671-4cab-add5-09cd26645207 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:09.512933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940d8365-9671-4cab-add5-09cd26645207 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:09.513507 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940d8365-9671-4cab-add5-09cd26645207 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:09.513962 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940d8365-9671-4cab-add5-09cd26645207 admin admin] PATH_INFO: `/v3/users/ce9ce2d0d06f4b059d05d68f588c9e66` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:09.515327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940d8365-9671-4cab-add5-09cd26645207 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ce9ce2d0d06f4b059d05d68f588c9e66)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:09.517561 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940d8365-9671-4cab-add5-09cd26645207 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:09.584096 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8711a5af-27c0-486b-8e94-ea6e11daa603 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:09.585827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8711a5af-27c0-486b-8e94-ea6e11daa603 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:09.618855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8711a5af-27c0-486b-8e94-ea6e11daa603 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:09.632323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:09.633094 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:09.633407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:09.633888 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe admin admin] PATH_INFO: `/v3/projects/ca0931d51497476f9f229632aa8e227b` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:09.635096 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe admin admin] RBAC: Authorizing `identity:delete_project(project_id=ca0931d51497476f9f229632aa8e227b)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:09.636982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:09.721874 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-940d8365-9671-4cab-add5-09cd26645207 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ce9ce2d0d06f4b059d05d68f588c9e66'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:09.764353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ca0931d51497476f9f229632aa8e227b', 'request_id': 'req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:09.775205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ca0931d51497476f9f229632aa8e227b', 'request_id': 'req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:09.791775 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-940d8365-9671-4cab-add5-09cd26645207 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ce9ce2d0d06f4b059d05d68f588c9e66'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:09.808125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-940d8365-9671-4cab-add5-09cd26645207 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ce9ce2d0d06f4b059d05d68f588c9e66'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:09.808922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-940d8365-9671-4cab-add5-09cd26645207 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ce9ce2d0d06f4b059d05d68f588c9e66'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:09.828509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-940d8365-9671-4cab-add5-09cd26645207 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ce9ce2d0d06f4b059d05d68f588c9e66'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:09.828509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 970/1930] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:09 2022] DELETE /identity/v3/users/ce9ce2d0d06f4b059d05d68f588c9e66 => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:09.854547 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ca0931d51497476f9f229632aa8e227b', 'request_id': 'req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:09.870790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe admin admin] The token cache is being invalidate because project ca0931d51497476f9f229632aa8e227b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:09.871419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:09.872407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:09.874704 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 961/1931] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:09 2022] DELETE /identity/v3/projects/ca0931d51497476f9f229632aa8e227b => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:06:09.880254 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940d8365-9671-4cab-add5-09cd26645207 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:09.887275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-940d8365-9671-4cab-add5-09cd26645207 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:09.983211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940d8365-9671-4cab-add5-09cd26645207 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:09.991156 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2075242-4927-4406-9a5f-1588ea0122d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:09.993270 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2075242-4927-4406-9a5f-1588ea0122d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:09.993270 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2075242-4927-4406-9a5f-1588ea0122d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:09.993270 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2075242-4927-4406-9a5f-1588ea0122d5 admin admin] PATH_INFO: `/v3/users/aab386fa32db456a97ad1a1966b8d24d` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:09.995302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2075242-4927-4406-9a5f-1588ea0122d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aab386fa32db456a97ad1a1966b8d24d)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:09.996422 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2075242-4927-4406-9a5f-1588ea0122d5 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:10.222750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-d2075242-4927-4406-9a5f-1588ea0122d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aab386fa32db456a97ad1a1966b8d24d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:10.296328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:10.297988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:10.310786 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:10.338794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-d2075242-4927-4406-9a5f-1588ea0122d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aab386fa32db456a97ad1a1966b8d24d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:10.354038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-d2075242-4927-4406-9a5f-1588ea0122d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aab386fa32db456a97ad1a1966b8d24d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:10.370446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:10.371815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e810c913-9dd4-4374-a4ff-13f2bc0d6fbe admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:10.373133 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-d2075242-4927-4406-9a5f-1588ea0122d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aab386fa32db456a97ad1a1966b8d24d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:10.377008 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88eaf9c8-ecf8-48a4-916d-b735bf47a56d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:10.377714 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eaf9c8-ecf8-48a4-916d-b735bf47a56d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:10.378628 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eaf9c8-ecf8-48a4-916d-b735bf47a56d admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:10.378980 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eaf9c8-ecf8-48a4-916d-b735bf47a56d admin admin] PATH_INFO: `/v3/users/5d1a4360c16d4b49a59baf59622b6d16` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:10.380721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88eaf9c8-ecf8-48a4-916d-b735bf47a56d admin admin] RBAC: Authorizing `identity:delete_user(user_id=5d1a4360c16d4b49a59baf59622b6d16)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:10.383133 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88eaf9c8-ecf8-48a4-916d-b735bf47a56d admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:10.402170 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-d2075242-4927-4406-9a5f-1588ea0122d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aab386fa32db456a97ad1a1966b8d24d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:10.412547 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 971/1932] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:09 2022] DELETE /identity/v3/users/aab386fa32db456a97ad1a1966b8d24d => generated 0 bytes in 530 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:10.413552 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2075242-4927-4406-9a5f-1588ea0122d5 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:10.415168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d2075242-4927-4406-9a5f-1588ea0122d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:10.526783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2075242-4927-4406-9a5f-1588ea0122d5 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:10.540581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6658fa13-433a-47b0-a7df-418b39b40e3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:10.541332 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6658fa13-433a-47b0-a7df-418b39b40e3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:10.541619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6658fa13-433a-47b0-a7df-418b39b40e3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:10.542048 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6658fa13-433a-47b0-a7df-418b39b40e3f admin admin] PATH_INFO: `/v3/projects/4612d1d746474c4ea901ab702b744f99` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:10.543887 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6658fa13-433a-47b0-a7df-418b39b40e3f admin admin] RBAC: Authorizing `identity:delete_project(project_id=4612d1d746474c4ea901ab702b744f99)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:10.546239 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6658fa13-433a-47b0-a7df-418b39b40e3f admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:10.622341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-88eaf9c8-ecf8-48a4-916d-b735bf47a56d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5d1a4360c16d4b49a59baf59622b6d16'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:10.623404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-88eaf9c8-ecf8-48a4-916d-b735bf47a56d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5d1a4360c16d4b49a59baf59622b6d16'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:10.705261 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-6658fa13-433a-47b0-a7df-418b39b40e3f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4612d1d746474c4ea901ab702b744f99', 'request_id': 'req-6658fa13-433a-47b0-a7df-418b39b40e3f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:10.722942 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-6658fa13-433a-47b0-a7df-418b39b40e3f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4612d1d746474c4ea901ab702b744f99', 'request_id': 'req-6658fa13-433a-47b0-a7df-418b39b40e3f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:10.729828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-6658fa13-433a-47b0-a7df-418b39b40e3f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4612d1d746474c4ea901ab702b744f99', 'request_id': 'req-6658fa13-433a-47b0-a7df-418b39b40e3f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:10.760618 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-88eaf9c8-ecf8-48a4-916d-b735bf47a56d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5d1a4360c16d4b49a59baf59622b6d16'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:10.776075 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-88eaf9c8-ecf8-48a4-916d-b735bf47a56d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5d1a4360c16d4b49a59baf59622b6d16'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:10.776850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-88eaf9c8-ecf8-48a4-916d-b735bf47a56d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5d1a4360c16d4b49a59baf59622b6d16'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:10.791210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 962/1933] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:10 2022] DELETE /identity/v3/users/5d1a4360c16d4b49a59baf59622b6d16 => generated 0 bytes in 495 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:06:10.796904 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7eb689-d67f-4449-9bbc-1e68ce6223e7 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:10.797519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7eb689-d67f-4449-9bbc-1e68ce6223e7 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:10.797856 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7eb689-d67f-4449-9bbc-1e68ce6223e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:10.823359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-fa7eb689-d67f-4449-9bbc-1e68ce6223e7 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:10.824608 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-fa7eb689-d67f-4449-9bbc-1e68ce6223e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:10.846399 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-6658fa13-433a-47b0-a7df-418b39b40e3f admin admin] The token cache is being invalidate because project 4612d1d746474c4ea901ab702b744f99 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:10.847815 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-6658fa13-433a-47b0-a7df-418b39b40e3f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:10.848621 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-6658fa13-433a-47b0-a7df-418b39b40e3f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:10.850667 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 972/1934] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:10 2022] DELETE /identity/v3/projects/4612d1d746474c4ea901ab702b744f99 => generated 0 bytes in 437 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:10.866479 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b445b70a-73fa-46f9-acfc-7951fbaf97bd None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:10.866758 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b445b70a-73fa-46f9-acfc-7951fbaf97bd None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:10.867420 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b445b70a-73fa-46f9-acfc-7951fbaf97bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:10.898616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-b445b70a-73fa-46f9-acfc-7951fbaf97bd None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:10.899875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b445b70a-73fa-46f9-acfc-7951fbaf97bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:10.924028 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 963/1935] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:10.934826 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7eb689-d67f-4449-9bbc-1e68ce6223e7 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:10.936160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-fa7eb689-d67f-4449-9bbc-1e68ce6223e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:10.987735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 973/1936] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:06:10.988491 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b445b70a-73fa-46f9-acfc-7951fbaf97bd None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:10.989857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b445b70a-73fa-46f9-acfc-7951fbaf97bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:11.007017 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b445b70a-73fa-46f9-acfc-7951fbaf97bd None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:11.012348 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f258a8-283c-43ff-9790-2f2ddbe6914d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:11.013098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f258a8-283c-43ff-9790-2f2ddbe6914d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:11.013487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f258a8-283c-43ff-9790-2f2ddbe6914d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:11.013831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f258a8-283c-43ff-9790-2f2ddbe6914d admin admin] PATH_INFO: `/v3/projects/1e621e0c178344bd86a1cd86734c629f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:11.015068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f258a8-283c-43ff-9790-2f2ddbe6914d admin admin] RBAC: Authorizing `identity:delete_project(project_id=1e621e0c178344bd86a1cd86734c629f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:11.017351 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f258a8-283c-43ff-9790-2f2ddbe6914d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:11.018258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7eb689-d67f-4449-9bbc-1e68ce6223e7 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:11.023389 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8098b7f1-ff3a-4e20-b9ba-8dbdd82a3bd2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:11.024010 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8098b7f1-ff3a-4e20-b9ba-8dbdd82a3bd2 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:11.024287 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8098b7f1-ff3a-4e20-b9ba-8dbdd82a3bd2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:11.024555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8098b7f1-ff3a-4e20-b9ba-8dbdd82a3bd2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:11.038527 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8098b7f1-ff3a-4e20-b9ba-8dbdd82a3bd2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:11.039780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8098b7f1-ff3a-4e20-b9ba-8dbdd82a3bd2 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:11.051246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 964/1937] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:06:11.052726 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad590b77-eae5-4fae-9d12-946d2ae820ec None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:11.053019 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad590b77-eae5-4fae-9d12-946d2ae820ec None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:11.053425 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad590b77-eae5-4fae-9d12-946d2ae820ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:11.074137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-ad590b77-eae5-4fae-9d12-946d2ae820ec None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:11.075364 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ad590b77-eae5-4fae-9d12-946d2ae820ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:11.084923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 965/1938] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:11.089404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad590b77-eae5-4fae-9d12-946d2ae820ec None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:11.100503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad590b77-eae5-4fae-9d12-946d2ae820ec None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:11.106070 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fb3c8d-7057-4eaa-b952-d0790a17c99a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:11.106690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fb3c8d-7057-4eaa-b952-d0790a17c99a admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:11.107315 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fb3c8d-7057-4eaa-b952-d0790a17c99a admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:11.107635 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fb3c8d-7057-4eaa-b952-d0790a17c99a admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:11.108436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fb3c8d-7057-4eaa-b952-d0790a17c99a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:06:11.108741 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fb3c8d-7057-4eaa-b952-d0790a17c99a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:11.110387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fb3c8d-7057-4eaa-b952-d0790a17c99a admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:11.132466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 966/1939] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:06:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 01:06:11.133689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fb3c8d-7057-4eaa-b952-d0790a17c99a admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:11.150090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fb3c8d-7057-4eaa-b952-d0790a17c99a admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:11.159997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-89f258a8-283c-43ff-9790-2f2ddbe6914d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1e621e0c178344bd86a1cd86734c629f', 'request_id': 'req-89f258a8-283c-43ff-9790-2f2ddbe6914d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:11.172987 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-89f258a8-283c-43ff-9790-2f2ddbe6914d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1e621e0c178344bd86a1cd86734c629f', 'request_id': 'req-89f258a8-283c-43ff-9790-2f2ddbe6914d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:11.186887 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-89f258a8-283c-43ff-9790-2f2ddbe6914d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1e621e0c178344bd86a1cd86734c629f', 'request_id': 'req-89f258a8-283c-43ff-9790-2f2ddbe6914d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:11.227674 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7a8e89-e9da-428d-8e72-d0efab76be45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:11.235346 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7a8e89-e9da-428d-8e72-d0efab76be45 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:11.235346 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7a8e89-e9da-428d-8e72-d0efab76be45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:11.235346 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7a8e89-e9da-428d-8e72-d0efab76be45 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:11.235346 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7a8e89-e9da-428d-8e72-d0efab76be45 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:11.235346 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7a8e89-e9da-428d-8e72-d0efab76be45 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:11.260916 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-89f258a8-283c-43ff-9790-2f2ddbe6914d admin admin] The token cache is being invalidate because project 1e621e0c178344bd86a1cd86734c629f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:11.261382 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-89f258a8-283c-43ff-9790-2f2ddbe6914d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:11.262065 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-89f258a8-283c-43ff-9790-2f2ddbe6914d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:11.263605 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 974/1940] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:10 2022] DELETE /identity/v3/projects/1e621e0c178344bd86a1cd86734c629f => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:11.268355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fb1ba9-30d6-4ccd-bac9-cfa743931813 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:11.269280 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fb1ba9-30d6-4ccd-bac9-cfa743931813 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:11.269280 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fb1ba9-30d6-4ccd-bac9-cfa743931813 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:11.293664 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-d3fb1ba9-30d6-4ccd-bac9-cfa743931813 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:11.294782 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d3fb1ba9-30d6-4ccd-bac9-cfa743931813 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:11.309640 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 975/1941] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:11.314313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d409c68d-7fee-4cd0-8cff-f0aba86c0bfb None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:11.314587 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d409c68d-7fee-4cd0-8cff-f0aba86c0bfb None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:11.314864 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d409c68d-7fee-4cd0-8cff-f0aba86c0bfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:11.344742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-d409c68d-7fee-4cd0-8cff-f0aba86c0bfb None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:11.346143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d409c68d-7fee-4cd0-8cff-f0aba86c0bfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:11.374600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 967/1942] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:06:11 2022] POST /identity/v3/projects => generated 375 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:06:11.375885 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7a8e89-e9da-428d-8e72-d0efab76be45 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:11.380898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ab7a8e89-e9da-428d-8e72-d0efab76be45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:11.387049 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 976/1943] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:06:11.389420 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d409c68d-7fee-4cd0-8cff-f0aba86c0bfb None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:11.415488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d409c68d-7fee-4cd0-8cff-f0aba86c0bfb None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:11.510340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:11.540613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14a2e38-6176-4137-ba03-56323f0a3ba7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:11.544704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a2e38-6176-4137-ba03-56323f0a3ba7 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:11.545924 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a2e38-6176-4137-ba03-56323f0a3ba7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:11.546366 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a2e38-6176-4137-ba03-56323f0a3ba7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:11.548729 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14a2e38-6176-4137-ba03-56323f0a3ba7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:06:11.549622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14a2e38-6176-4137-ba03-56323f0a3ba7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:11.553517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14a2e38-6176-4137-ba03-56323f0a3ba7 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:11.567310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7a8e89-e9da-428d-8e72-d0efab76be45 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:11.568168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:11.580374 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35630f06-2d1a-4cd9-9fff-0ef3b2595589 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:11.583330 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35630f06-2d1a-4cd9-9fff-0ef3b2595589 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:11.583621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35630f06-2d1a-4cd9-9fff-0ef3b2595589 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:11.583902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35630f06-2d1a-4cd9-9fff-0ef3b2595589 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:11.585729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35630f06-2d1a-4cd9-9fff-0ef3b2595589 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:11.586416 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 977/1944] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:06:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:06:11.587702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35630f06-2d1a-4cd9-9fff-0ef3b2595589 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:11.590522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14a2e38-6176-4137-ba03-56323f0a3ba7 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:11.603450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14a2e38-6176-4137-ba03-56323f0a3ba7 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:11.608889 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c2b047-0c50-47b1-b300-dc72d3a6102b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:11.609824 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c2b047-0c50-47b1-b300-dc72d3a6102b admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:11.610341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c2b047-0c50-47b1-b300-dc72d3a6102b admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:11.611064 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c2b047-0c50-47b1-b300-dc72d3a6102b admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:11.611874 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c2b047-0c50-47b1-b300-dc72d3a6102b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:11.614458 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c2b047-0c50-47b1-b300-dc72d3a6102b admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:11.679506 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 968/1945] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:06:11 2022] POST /identity/v3/users => generated 288 bytes in 304 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:06:11.680466 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35630f06-2d1a-4cd9-9fff-0ef3b2595589 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:11.708019 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35630f06-2d1a-4cd9-9fff-0ef3b2595589 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:11.722364 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d00ce1-ef2f-4f45-a655-7ccaeaf8b9c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:11.723011 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d00ce1-ef2f-4f45-a655-7ccaeaf8b9c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:11.723956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d00ce1-ef2f-4f45-a655-7ccaeaf8b9c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:11.723956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d00ce1-ef2f-4f45-a655-7ccaeaf8b9c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:11.724803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d00ce1-ef2f-4f45-a655-7ccaeaf8b9c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:11.727991 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d00ce1-ef2f-4f45-a655-7ccaeaf8b9c4 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:11.747703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 969/1946] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:11 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:06:11.753386 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d00ce1-ef2f-4f45-a655-7ccaeaf8b9c4 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:11.772654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d00ce1-ef2f-4f45-a655-7ccaeaf8b9c4 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:11.774176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 978/1947] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:06:11 2022] POST /identity/v3/projects => generated 371 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:06:11.777177 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c2b047-0c50-47b1-b300-dc72d3a6102b admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:11.896070 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7764b49-c7c9-42d0-a117-de729191fdd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:11.898110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7764b49-c7c9-42d0-a117-de729191fdd8 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:11.898451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7764b49-c7c9-42d0-a117-de729191fdd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:11.898737 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7764b49-c7c9-42d0-a117-de729191fdd8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:11.900283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7764b49-c7c9-42d0-a117-de729191fdd8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:11.902393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7764b49-c7c9-42d0-a117-de729191fdd8 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:11.905439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c2b047-0c50-47b1-b300-dc72d3a6102b admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:11.913582 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845d5522-6f42-4b36-9bfd-58fd00b931d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:11.917455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845d5522-6f42-4b36-9bfd-58fd00b931d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:11.917771 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845d5522-6f42-4b36-9bfd-58fd00b931d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:11.918811 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845d5522-6f42-4b36-9bfd-58fd00b931d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:11.921668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 970/1948] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:11 2022] GET /identity/v3/roles => generated 1420 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:06:11.926351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7764b49-c7c9-42d0-a117-de729191fdd8 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:11.931589 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845d5522-6f42-4b36-9bfd-58fd00b931d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:11.939678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845d5522-6f42-4b36-9bfd-58fd00b931d4 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:11.940009 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7764b49-c7c9-42d0-a117-de729191fdd8 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:11.948066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f003d9b2-6c88-4cc6-9b3b-0098fb7ae580 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:11.949130 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f003d9b2-6c88-4cc6-9b3b-0098fb7ae580 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:11.949932 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f003d9b2-6c88-4cc6-9b3b-0098fb7ae580 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:11.952204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f003d9b2-6c88-4cc6-9b3b-0098fb7ae580 admin admin] PATH_INFO: `/v3/projects/528d5c15cc47445dbe46761067a1c54a/users/af874cda15384e2f9160ccf6866296f6/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:11.999389 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f003d9b2-6c88-4cc6-9b3b-0098fb7ae580 admin admin] RBAC: Authorizing `identity:create_grant(project_id=528d5c15cc47445dbe46761067a1c54a, user_id=af874cda15384e2f9160ccf6866296f6, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:12.007448 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f003d9b2-6c88-4cc6-9b3b-0098fb7ae580 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:12.078076 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 979/1949] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:06:11 2022] POST /identity/v3/users => generated 286 bytes in 301 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:06:12.083288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845d5522-6f42-4b36-9bfd-58fd00b931d4 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:12.118901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 971/1950] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:06:11 2022] PUT /identity/v3/projects/528d5c15cc47445dbe46761067a1c54a/users/af874cda15384e2f9160ccf6866296f6/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:06:12.119366 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845d5522-6f42-4b36-9bfd-58fd00b931d4 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:12.121762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f766b8-d745-45d3-a006-0fc76218e3f9 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:12.121762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f766b8-d745-45d3-a006-0fc76218e3f9 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:12.121762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f766b8-d745-45d3-a006-0fc76218e3f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:12.193723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-10f766b8-d745-45d3-a006-0fc76218e3f9 None None] MFA Rules not processed for user `af874cda15384e2f9160ccf6866296f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:12.195225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-10f766b8-d745-45d3-a006-0fc76218e3f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:12.297330 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f22f22-2f42-4456-8cbd-9825c97d72a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:12.298114 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f22f22-2f42-4456-8cbd-9825c97d72a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:12.298445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f22f22-2f42-4456-8cbd-9825c97d72a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:12.299580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f22f22-2f42-4456-8cbd-9825c97d72a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:12.300717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f22f22-2f42-4456-8cbd-9825c97d72a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:12.302468 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f22f22-2f42-4456-8cbd-9825c97d72a8 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:12.305968 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 972/1951] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:12 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Sep 04 01:06:12.311700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f766b8-d745-45d3-a006-0fc76218e3f9 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:12.313285 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-10f766b8-d745-45d3-a006-0fc76218e3f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:12.315162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 980/1952] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:12 2022] GET /identity/v3/roles => generated 1420 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:06:12.321481 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f22f22-2f42-4456-8cbd-9825c97d72a8 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:12.335463 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f22f22-2f42-4456-8cbd-9825c97d72a8 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:12.335734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f766b8-d745-45d3-a006-0fc76218e3f9 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:12.341629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a74804f-d8e2-44f5-ac80-ba4a55acc237 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:12.342281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a74804f-d8e2-44f5-ac80-ba4a55acc237 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:12.343070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a74804f-d8e2-44f5-ac80-ba4a55acc237 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:12.343791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78caf063-2694-41b3-8c4b-98ce8e0a98e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:12.343942 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a74804f-d8e2-44f5-ac80-ba4a55acc237 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:12.345013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78caf063-2694-41b3-8c4b-98ce8e0a98e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:12.345165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a74804f-d8e2-44f5-ac80-ba4a55acc237 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:12.345819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78caf063-2694-41b3-8c4b-98ce8e0a98e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:12.346317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78caf063-2694-41b3-8c4b-98ce8e0a98e9 admin admin] PATH_INFO: `/v3/projects/5d8ec7c6ee57418eb3a95d2e0b857f75` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:12.347875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a74804f-d8e2-44f5-ac80-ba4a55acc237 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:12.349155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78caf063-2694-41b3-8c4b-98ce8e0a98e9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d8ec7c6ee57418eb3a95d2e0b857f75)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:12.351421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78caf063-2694-41b3-8c4b-98ce8e0a98e9 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:12.361840 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 981/1953] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:12 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:06:12.361840 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a74804f-d8e2-44f5-ac80-ba4a55acc237 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:12.368369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-6a74804f-d8e2-44f5-ac80-ba4a55acc237 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:12.450286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a74804f-d8e2-44f5-ac80-ba4a55acc237 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:12.456853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e519d79a-93a6-463c-a10b-53c313f59d6f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:12.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e519d79a-93a6-463c-a10b-53c313f59d6f service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:12.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e519d79a-93a6-463c-a10b-53c313f59d6f service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:12.465109 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e519d79a-93a6-463c-a10b-53c313f59d6f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:12.477746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e519d79a-93a6-463c-a10b-53c313f59d6f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:12.479304 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e519d79a-93a6-463c-a10b-53c313f59d6f service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:12.492370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 982/1954] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:12 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:12.493150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e519d79a-93a6-463c-a10b-53c313f59d6f service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:12.505231 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e519d79a-93a6-463c-a10b-53c313f59d6f service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:12.507320 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-78caf063-2694-41b3-8c4b-98ce8e0a98e9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d8ec7c6ee57418eb3a95d2e0b857f75', 'request_id': 'req-78caf063-2694-41b3-8c4b-98ce8e0a98e9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:12.514463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-78caf063-2694-41b3-8c4b-98ce8e0a98e9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d8ec7c6ee57418eb3a95d2e0b857f75', 'request_id': 'req-78caf063-2694-41b3-8c4b-98ce8e0a98e9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:12.573252 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a522c3-f0c1-4338-8da8-a230446436a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:12.574112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a522c3-f0c1-4338-8da8-a230446436a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:12.574430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a522c3-f0c1-4338-8da8-a230446436a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:12.574710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a522c3-f0c1-4338-8da8-a230446436a4 admin admin] PATH_INFO: `/v3/projects/e53f2763660041c5909c9e61a27f078c/users/c488bcdb6a584cc0a756d08b4597578e/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:12.591819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a522c3-f0c1-4338-8da8-a230446436a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e53f2763660041c5909c9e61a27f078c, user_id=c488bcdb6a584cc0a756d08b4597578e, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:12.594577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a522c3-f0c1-4338-8da8-a230446436a4 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:12.602375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:12.603631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-78caf063-2694-41b3-8c4b-98ce8e0a98e9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d8ec7c6ee57418eb3a95d2e0b857f75', 'request_id': 'req-78caf063-2694-41b3-8c4b-98ce8e0a98e9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:12.620407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-78caf063-2694-41b3-8c4b-98ce8e0a98e9 admin admin] The token cache is being invalidate because project 5d8ec7c6ee57418eb3a95d2e0b857f75 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:12.620730 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-78caf063-2694-41b3-8c4b-98ce8e0a98e9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:12.621489 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-78caf063-2694-41b3-8c4b-98ce8e0a98e9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:12.624352 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 973/1955] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:12 2022] DELETE /identity/v3/projects/5d8ec7c6ee57418eb3a95d2e0b857f75 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:12.629644 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78caf063-2694-41b3-8c4b-98ce8e0a98e9 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:12.631498 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-78caf063-2694-41b3-8c4b-98ce8e0a98e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:12.647054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78caf063-2694-41b3-8c4b-98ce8e0a98e9 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:12.654612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ea2ad3-0b40-4d5f-a2e7-74e925a3fd4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:12.654820 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ea2ad3-0b40-4d5f-a2e7-74e925a3fd4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:12.655142 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ea2ad3-0b40-4d5f-a2e7-74e925a3fd4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:12.655631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ea2ad3-0b40-4d5f-a2e7-74e925a3fd4d admin admin] PATH_INFO: `/v3/users/8e0207d52df145e0b44afe10d59f8ad6` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:12.675960 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 983/1956] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:06:12 2022] PUT /identity/v3/projects/e53f2763660041c5909c9e61a27f078c/users/c488bcdb6a584cc0a756d08b4597578e/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:12.677683 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717e393f-ce15-4328-b3e6-d6cf6f72c2b9 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:12.678735 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ea2ad3-0b40-4d5f-a2e7-74e925a3fd4d admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e0207d52df145e0b44afe10d59f8ad6)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:12.679077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717e393f-ce15-4328-b3e6-d6cf6f72c2b9 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:12.679434 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717e393f-ce15-4328-b3e6-d6cf6f72c2b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:12.684135 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ea2ad3-0b40-4d5f-a2e7-74e925a3fd4d admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:12.719767 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-717e393f-ce15-4328-b3e6-d6cf6f72c2b9 None None] MFA Rules not processed for user `c488bcdb6a584cc0a756d08b4597578e`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:12.721320 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-717e393f-ce15-4328-b3e6-d6cf6f72c2b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:12.759619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:12.813873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 984/1957] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:12 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:12.838199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717e393f-ce15-4328-b3e6-d6cf6f72c2b9 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:12.840724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-717e393f-ce15-4328-b3e6-d6cf6f72c2b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:12.916943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717e393f-ce15-4328-b3e6-d6cf6f72c2b9 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:12.921334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa1cde2-f2f4-46a9-a9a4-683540f91cf7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:12.922174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa1cde2-f2f4-46a9-a9a4-683540f91cf7 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:12.922607 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa1cde2-f2f4-46a9-a9a4-683540f91cf7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:12.923325 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa1cde2-f2f4-46a9-a9a4-683540f91cf7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:12.927542 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d1ea2ad3-0b40-4d5f-a2e7-74e925a3fd4d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e0207d52df145e0b44afe10d59f8ad6'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:12.928314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d1ea2ad3-0b40-4d5f-a2e7-74e925a3fd4d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e0207d52df145e0b44afe10d59f8ad6'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:12.937771 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa1cde2-f2f4-46a9-a9a4-683540f91cf7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:12.939471 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa1cde2-f2f4-46a9-a9a4-683540f91cf7 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:12.949870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 985/1958] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:12 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:12.974719 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa1cde2-f2f4-46a9-a9a4-683540f91cf7 service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:12.974719 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-aaa1cde2-f2f4-46a9-a9a4-683540f91cf7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:13.023533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d1ea2ad3-0b40-4d5f-a2e7-74e925a3fd4d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e0207d52df145e0b44afe10d59f8ad6'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:13.044686 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d1ea2ad3-0b40-4d5f-a2e7-74e925a3fd4d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e0207d52df145e0b44afe10d59f8ad6'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:13.045529 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d1ea2ad3-0b40-4d5f-a2e7-74e925a3fd4d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e0207d52df145e0b44afe10d59f8ad6'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:13.060614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 974/1959] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:12 2022] DELETE /identity/v3/users/8e0207d52df145e0b44afe10d59f8ad6 => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:13.085511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa1cde2-f2f4-46a9-a9a4-683540f91cf7 service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:13.166724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648d607f-3de3-41ed-a5f4-122b0bbb0d6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:13.166724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648d607f-3de3-41ed-a5f4-122b0bbb0d6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:13.166724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648d607f-3de3-41ed-a5f4-122b0bbb0d6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:13.166724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648d607f-3de3-41ed-a5f4-122b0bbb0d6e admin admin] PATH_INFO: `/v3/users/c488bcdb6a584cc0a756d08b4597578e` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:13.167195 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648d607f-3de3-41ed-a5f4-122b0bbb0d6e admin admin] RBAC: Authorizing `identity:delete_user(user_id=c488bcdb6a584cc0a756d08b4597578e)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:13.169226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648d607f-3de3-41ed-a5f4-122b0bbb0d6e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:13.467478 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-648d607f-3de3-41ed-a5f4-122b0bbb0d6e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c488bcdb6a584cc0a756d08b4597578e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:13.571665 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-648d607f-3de3-41ed-a5f4-122b0bbb0d6e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c488bcdb6a584cc0a756d08b4597578e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:13.585136 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-648d607f-3de3-41ed-a5f4-122b0bbb0d6e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c488bcdb6a584cc0a756d08b4597578e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:13.585869 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-648d607f-3de3-41ed-a5f4-122b0bbb0d6e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c488bcdb6a584cc0a756d08b4597578e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:13.592937 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-648d607f-3de3-41ed-a5f4-122b0bbb0d6e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c488bcdb6a584cc0a756d08b4597578e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:13.600525 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 986/1960] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:12 2022] DELETE /identity/v3/users/c488bcdb6a584cc0a756d08b4597578e => generated 0 bytes in 629 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:06:13.608438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ea2ad3-0b40-4d5f-a2e7-74e925a3fd4d admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:13.610191 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5790b45-cc82-4e27-b636-a2a988bf9446 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:13.610940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5790b45-cc82-4e27-b636-a2a988bf9446 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:13.611228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5790b45-cc82-4e27-b636-a2a988bf9446 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:13.611647 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d1ea2ad3-0b40-4d5f-a2e7-74e925a3fd4d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:13.631455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-b5790b45-cc82-4e27-b636-a2a988bf9446 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:13.638850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ea2ad3-0b40-4d5f-a2e7-74e925a3fd4d admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:13.639458 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b5790b45-cc82-4e27-b636-a2a988bf9446 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:13.690565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d85bb6-8367-44f8-bda4-bcf930a9d918 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:13.691202 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d85bb6-8367-44f8-bda4-bcf930a9d918 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:13.691718 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d85bb6-8367-44f8-bda4-bcf930a9d918 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:13.692069 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d85bb6-8367-44f8-bda4-bcf930a9d918 admin admin] PATH_INFO: `/v3/projects/d30baef84b62499fac7cfe20946356ec` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:13.693366 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d85bb6-8367-44f8-bda4-bcf930a9d918 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d30baef84b62499fac7cfe20946356ec)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:13.695430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 987/1961] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:13.696315 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d85bb6-8367-44f8-bda4-bcf930a9d918 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:13.733214 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5790b45-cc82-4e27-b636-a2a988bf9446 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:13.734643 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b5790b45-cc82-4e27-b636-a2a988bf9446 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:13.794527 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5790b45-cc82-4e27-b636-a2a988bf9446 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:13.811674 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a9d85bb6-8367-44f8-bda4-bcf930a9d918 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd30baef84b62499fac7cfe20946356ec', 'request_id': 'req-a9d85bb6-8367-44f8-bda4-bcf930a9d918'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:13.818791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a9d85bb6-8367-44f8-bda4-bcf930a9d918 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd30baef84b62499fac7cfe20946356ec', 'request_id': 'req-a9d85bb6-8367-44f8-bda4-bcf930a9d918'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:13.845866 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57db2610-08ea-44b1-8e77-f53bf2d763f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:13.846406 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57db2610-08ea-44b1-8e77-f53bf2d763f1 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:13.847255 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57db2610-08ea-44b1-8e77-f53bf2d763f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:13.847711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57db2610-08ea-44b1-8e77-f53bf2d763f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:13.888244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a9d85bb6-8367-44f8-bda4-bcf930a9d918 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd30baef84b62499fac7cfe20946356ec', 'request_id': 'req-a9d85bb6-8367-44f8-bda4-bcf930a9d918'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:13.904024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a9d85bb6-8367-44f8-bda4-bcf930a9d918 admin admin] The token cache is being invalidate because project d30baef84b62499fac7cfe20946356ec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:13.904593 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a9d85bb6-8367-44f8-bda4-bcf930a9d918 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:13.905298 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a9d85bb6-8367-44f8-bda4-bcf930a9d918 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:13.907237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 975/1962] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:13 2022] DELETE /identity/v3/projects/d30baef84b62499fac7cfe20946356ec => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:13.912812 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70159313-6c1a-449c-afe7-403f91e5c887 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:13.913683 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70159313-6c1a-449c-afe7-403f91e5c887 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:13.914294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70159313-6c1a-449c-afe7-403f91e5c887 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:13.922313 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:13.941889 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57db2610-08ea-44b1-8e77-f53bf2d763f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:13.942671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-70159313-6c1a-449c-afe7-403f91e5c887 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:13.943654 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57db2610-08ea-44b1-8e77-f53bf2d763f1 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:13.944857 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-70159313-6c1a-449c-afe7-403f91e5c887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:13.957577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 976/1963] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:13.958334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 988/1964] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:13.966317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af0bd81-e7d9-4096-98c9-981dcecc7fb8 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:13.966816 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af0bd81-e7d9-4096-98c9-981dcecc7fb8 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:13.967243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af0bd81-e7d9-4096-98c9-981dcecc7fb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:13.991889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-6af0bd81-e7d9-4096-98c9-981dcecc7fb8 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:13.993245 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6af0bd81-e7d9-4096-98c9-981dcecc7fb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:14.004817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 977/1965] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:14.005980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57db2610-08ea-44b1-8e77-f53bf2d763f1 service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:14.007330 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-57db2610-08ea-44b1-8e77-f53bf2d763f1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:14.021620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57db2610-08ea-44b1-8e77-f53bf2d763f1 service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:14.028709 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcce7aa-144d-40a1-a985-a892012b9eac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:14.028887 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcce7aa-144d-40a1-a985-a892012b9eac admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:14.029152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcce7aa-144d-40a1-a985-a892012b9eac admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:14.029415 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcce7aa-144d-40a1-a985-a892012b9eac admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:14.030076 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcce7aa-144d-40a1-a985-a892012b9eac admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:06:14.030900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcce7aa-144d-40a1-a985-a892012b9eac admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:14.032732 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcce7aa-144d-40a1-a985-a892012b9eac admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:14.046205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 989/1966] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:06:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:06:14.050936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af0bd81-e7d9-4096-98c9-981dcecc7fb8 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:14.060436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af0bd81-e7d9-4096-98c9-981dcecc7fb8 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:14.066399 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4454fea9-11af-4d60-abe1-48b7fd55a7f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:14.067309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4454fea9-11af-4d60-abe1-48b7fd55a7f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:14.067594 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4454fea9-11af-4d60-abe1-48b7fd55a7f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:14.068030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4454fea9-11af-4d60-abe1-48b7fd55a7f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:14.068676 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4454fea9-11af-4d60-abe1-48b7fd55a7f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:14.071160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4454fea9-11af-4d60-abe1-48b7fd55a7f1 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:14.175132 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 978/1967] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:06:14 2022] POST /identity/v3/projects => generated 383 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:06:14.179491 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcce7aa-144d-40a1-a985-a892012b9eac admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:14.201019 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcce7aa-144d-40a1-a985-a892012b9eac admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:14.279007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329205d8-b722-4682-98ea-2fefbc533405 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:14.279671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329205d8-b722-4682-98ea-2fefbc533405 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:14.279979 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329205d8-b722-4682-98ea-2fefbc533405 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:14.280697 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329205d8-b722-4682-98ea-2fefbc533405 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:14.281054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329205d8-b722-4682-98ea-2fefbc533405 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:14.283050 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329205d8-b722-4682-98ea-2fefbc533405 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:14.323418 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:14.364874 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 990/1968] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:06:14 2022] POST /identity/v3/users => generated 292 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:06:14.365722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4454fea9-11af-4d60-abe1-48b7fd55a7f1 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:14.379208 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4454fea9-11af-4d60-abe1-48b7fd55a7f1 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:14.395233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f55d19e-8bee-4df0-a0f5-dbd586fac20a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:14.395872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f55d19e-8bee-4df0-a0f5-dbd586fac20a admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:14.396149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f55d19e-8bee-4df0-a0f5-dbd586fac20a admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:14.396417 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f55d19e-8bee-4df0-a0f5-dbd586fac20a admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:14.407858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f55d19e-8bee-4df0-a0f5-dbd586fac20a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:14.410108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f55d19e-8bee-4df0-a0f5-dbd586fac20a admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:14.429607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 979/1969] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:14 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:06:14.430912 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329205d8-b722-4682-98ea-2fefbc533405 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:14.444861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329205d8-b722-4682-98ea-2fefbc533405 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:14.482520 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d279db4-8d72-496e-b9ff-16b2dbe360dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:14.483193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d279db4-8d72-496e-b9ff-16b2dbe360dc admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:14.484452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d279db4-8d72-496e-b9ff-16b2dbe360dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:14.485114 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d279db4-8d72-496e-b9ff-16b2dbe360dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:14.486454 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d279db4-8d72-496e-b9ff-16b2dbe360dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:14.486781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b135d4-b836-4b98-846f-6a7b9374a587 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:14.487070 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b135d4-b836-4b98-846f-6a7b9374a587 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:14.488145 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b135d4-b836-4b98-846f-6a7b9374a587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:14.488880 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d279db4-8d72-496e-b9ff-16b2dbe360dc admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:14.502833 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 991/1970] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:14 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:06:14.506246 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d279db4-8d72-496e-b9ff-16b2dbe360dc admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:14.510953 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-91b135d4-b836-4b98-846f-6a7b9374a587 None None] MFA Rules not processed for user `af874cda15384e2f9160ccf6866296f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:14.512193 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-91b135d4-b836-4b98-846f-6a7b9374a587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:14.519456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d279db4-8d72-496e-b9ff-16b2dbe360dc admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:14.524987 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f39f482-ab2f-4373-b541-80d4c8937be9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:14.525857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f39f482-ab2f-4373-b541-80d4c8937be9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:14.526196 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f39f482-ab2f-4373-b541-80d4c8937be9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:14.526536 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f39f482-ab2f-4373-b541-80d4c8937be9 admin admin] PATH_INFO: `/v3/projects/2bea56a7089f4fa189405ee93b0993e5/users/47fb156027d14eb6a014bc9b9c89e9f6/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:14.543616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f39f482-ab2f-4373-b541-80d4c8937be9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2bea56a7089f4fa189405ee93b0993e5, user_id=47fb156027d14eb6a014bc9b9c89e9f6, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:14.546350 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f39f482-ab2f-4373-b541-80d4c8937be9 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:14.572588 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 980/1971] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:14 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:06:14.643201 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b135d4-b836-4b98-846f-6a7b9374a587 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:14.644815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-91b135d4-b836-4b98-846f-6a7b9374a587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:14.664865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b135d4-b836-4b98-846f-6a7b9374a587 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:14.674674 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c8b06f-6880-475e-bdee-3f34039f3fec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:14.677327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c8b06f-6880-475e-bdee-3f34039f3fec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:14.677902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c8b06f-6880-475e-bdee-3f34039f3fec admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:14.678348 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c8b06f-6880-475e-bdee-3f34039f3fec admin admin] PATH_INFO: `/v3/projects/e53f2763660041c5909c9e61a27f078c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:14.680055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c8b06f-6880-475e-bdee-3f34039f3fec admin admin] RBAC: Authorizing `identity:delete_project(project_id=e53f2763660041c5909c9e61a27f078c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:14.682510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c8b06f-6880-475e-bdee-3f34039f3fec admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:14.692278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 992/1972] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:06:14 2022] PUT /identity/v3/projects/2bea56a7089f4fa189405ee93b0993e5/users/47fb156027d14eb6a014bc9b9c89e9f6/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:06:14.702499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138f2eb6-28c2-44c1-bb2f-27c2e45e53b6 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:14.702784 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138f2eb6-28c2-44c1-bb2f-27c2e45e53b6 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:14.703057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138f2eb6-28c2-44c1-bb2f-27c2e45e53b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:14.756135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-138f2eb6-28c2-44c1-bb2f-27c2e45e53b6 None None] MFA Rules not processed for user `47fb156027d14eb6a014bc9b9c89e9f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:14.757763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-138f2eb6-28c2-44c1-bb2f-27c2e45e53b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:14.837870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 993/1973] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:14 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:06:14.842106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138f2eb6-28c2-44c1-bb2f-27c2e45e53b6 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:14.850010 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-88c8b06f-6880-475e-bdee-3f34039f3fec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e53f2763660041c5909c9e61a27f078c', 'request_id': 'req-88c8b06f-6880-475e-bdee-3f34039f3fec'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:14.858375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-88c8b06f-6880-475e-bdee-3f34039f3fec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e53f2763660041c5909c9e61a27f078c', 'request_id': 'req-88c8b06f-6880-475e-bdee-3f34039f3fec'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:14.910464 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-88c8b06f-6880-475e-bdee-3f34039f3fec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e53f2763660041c5909c9e61a27f078c', 'request_id': 'req-88c8b06f-6880-475e-bdee-3f34039f3fec'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:14.923793 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-88c8b06f-6880-475e-bdee-3f34039f3fec admin admin] The token cache is being invalidate because project e53f2763660041c5909c9e61a27f078c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:14.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-88c8b06f-6880-475e-bdee-3f34039f3fec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:14.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-88c8b06f-6880-475e-bdee-3f34039f3fec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:14.927978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 981/1974] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:14 2022] DELETE /identity/v3/projects/e53f2763660041c5909c9e61a27f078c => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:06:14.931498 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110ffb56-80af-4270-ba68-7177ab01df27 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:14.931769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110ffb56-80af-4270-ba68-7177ab01df27 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:14.932243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110ffb56-80af-4270-ba68-7177ab01df27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:14.937133 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138f2eb6-28c2-44c1-bb2f-27c2e45e53b6 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:14.954191 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-110ffb56-80af-4270-ba68-7177ab01df27 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:14.956856 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-110ffb56-80af-4270-ba68-7177ab01df27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:15.021180 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc24ae01-c426-4e88-a8ca-038c7b5e8dd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:15.021947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc24ae01-c426-4e88-a8ca-038c7b5e8dd8 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:15.022065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:15.022556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc24ae01-c426-4e88-a8ca-038c7b5e8dd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:15.025201 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc24ae01-c426-4e88-a8ca-038c7b5e8dd8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:15.026629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc24ae01-c426-4e88-a8ca-038c7b5e8dd8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:15.029659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc24ae01-c426-4e88-a8ca-038c7b5e8dd8 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:15.054179 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 982/1975] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:15.062811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4cd5d1-93b6-4620-b86f-d6781a7599c8 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:15.063106 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4cd5d1-93b6-4620-b86f-d6781a7599c8 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:15.063511 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4cd5d1-93b6-4620-b86f-d6781a7599c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:15.070617 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:15.088090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-fb4cd5d1-93b6-4620-b86f-d6781a7599c8 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:15.089245 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-fb4cd5d1-93b6-4620-b86f-d6781a7599c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:15.102607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 983/1976] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:15.103807 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4cd5d1-93b6-4620-b86f-d6781a7599c8 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:15.115553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4cd5d1-93b6-4620-b86f-d6781a7599c8 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:15.120779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b482def0-d87e-4324-8d44-4718af17aca4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:15.121476 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b482def0-d87e-4324-8d44-4718af17aca4 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:15.121734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b482def0-d87e-4324-8d44-4718af17aca4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:15.122003 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b482def0-d87e-4324-8d44-4718af17aca4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:15.122712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b482def0-d87e-4324-8d44-4718af17aca4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:06:15.123151 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b482def0-d87e-4324-8d44-4718af17aca4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:15.125305 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b482def0-d87e-4324-8d44-4718af17aca4 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:15.139689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 984/1977] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:06:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:06:15.187828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b482def0-d87e-4324-8d44-4718af17aca4 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:15.187828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b482def0-d87e-4324-8d44-4718af17aca4 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:15.187828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451c8039-a560-4add-b93a-380dbebe2399 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:15.187828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451c8039-a560-4add-b93a-380dbebe2399 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:15.212186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451c8039-a560-4add-b93a-380dbebe2399 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:15.212186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451c8039-a560-4add-b93a-380dbebe2399 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:15.212186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451c8039-a560-4add-b93a-380dbebe2399 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:15.212186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451c8039-a560-4add-b93a-380dbebe2399 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:15.463176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 994/1978] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:06:14 2022] POST /identity/v3/projects => generated 385 bytes in 609 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:06:15.468238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc24ae01-c426-4e88-a8ca-038c7b5e8dd8 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:15.470006 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-bc24ae01-c426-4e88-a8ca-038c7b5e8dd8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:15.546876 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc24ae01-c426-4e88-a8ca-038c7b5e8dd8 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:15.552753 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57488aa0-30eb-4b28-a2b4-65b01f72918e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:15.553437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57488aa0-30eb-4b28-a2b4-65b01f72918e admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:15.553717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57488aa0-30eb-4b28-a2b4-65b01f72918e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:15.553988 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57488aa0-30eb-4b28-a2b4-65b01f72918e admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:15.554666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57488aa0-30eb-4b28-a2b4-65b01f72918e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:15.556650 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57488aa0-30eb-4b28-a2b4-65b01f72918e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:15.719778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 985/1979] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:06:15 2022] POST /identity/v3/projects => generated 385 bytes in 563 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:06:15.721119 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451c8039-a560-4add-b93a-380dbebe2399 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:15.745387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 995/1980] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:06:15 2022] POST /identity/v3/users => generated 299 bytes in 273 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:06:15.745961 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57488aa0-30eb-4b28-a2b4-65b01f72918e admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:15.770528 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57488aa0-30eb-4b28-a2b4-65b01f72918e admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:15.842701 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451c8039-a560-4add-b93a-380dbebe2399 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:15.851446 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d297ee00-5826-4cb6-8267-457f51f2258d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:15.852509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d297ee00-5826-4cb6-8267-457f51f2258d admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:15.853013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d297ee00-5826-4cb6-8267-457f51f2258d admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:15.853533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d297ee00-5826-4cb6-8267-457f51f2258d admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:15.854604 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d297ee00-5826-4cb6-8267-457f51f2258d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:15.858321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d297ee00-5826-4cb6-8267-457f51f2258d admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:15.874458 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f06a083-a661-416f-9ef4-1ed9fc6470ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:15.875113 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f06a083-a661-416f-9ef4-1ed9fc6470ef admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:15.875953 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f06a083-a661-416f-9ef4-1ed9fc6470ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:15.875953 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f06a083-a661-416f-9ef4-1ed9fc6470ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:15.876288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f06a083-a661-416f-9ef4-1ed9fc6470ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:15.881359 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f06a083-a661-416f-9ef4-1ed9fc6470ef admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:15.893149 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 996/1981] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:15 2022] GET /identity/v3/roles => generated 1420 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:06:15.894276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f06a083-a661-416f-9ef4-1ed9fc6470ef admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:15.904350 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f06a083-a661-416f-9ef4-1ed9fc6470ef admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:15.911579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68b888b-d611-4e2f-bd54-6e06e00f4673 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:15.916681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68b888b-d611-4e2f-bd54-6e06e00f4673 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:15.917305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68b888b-d611-4e2f-bd54-6e06e00f4673 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:15.917683 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68b888b-d611-4e2f-bd54-6e06e00f4673 admin admin] PATH_INFO: `/v3/projects/c968576dfbe64995bae40694095cb817/users/8d8e9735e1844b4382b8d797caae6e8c/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:15.934477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68b888b-d611-4e2f-bd54-6e06e00f4673 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c968576dfbe64995bae40694095cb817, user_id=8d8e9735e1844b4382b8d797caae6e8c, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:15.937354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68b888b-d611-4e2f-bd54-6e06e00f4673 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:15.983102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 986/1982] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:06:15 2022] POST /identity/v3/users => generated 293 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:06:15.986178 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d297ee00-5826-4cb6-8267-457f51f2258d admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:15.996964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d297ee00-5826-4cb6-8267-457f51f2258d admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:16.002236 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb99e9f5-2766-4011-b887-7fae76973be4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:16.003761 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb99e9f5-2766-4011-b887-7fae76973be4 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:16.004284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb99e9f5-2766-4011-b887-7fae76973be4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:16.004620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb99e9f5-2766-4011-b887-7fae76973be4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:16.005370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb99e9f5-2766-4011-b887-7fae76973be4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:16.007134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb99e9f5-2766-4011-b887-7fae76973be4 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:16.025118 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 987/1983] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:15 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:06:16.026555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb99e9f5-2766-4011-b887-7fae76973be4 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:16.036975 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb99e9f5-2766-4011-b887-7fae76973be4 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:16.042162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6c2e33-feb9-4930-8ad7-8adb04aaf386 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:16.044480 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6c2e33-feb9-4930-8ad7-8adb04aaf386 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:16.044480 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6c2e33-feb9-4930-8ad7-8adb04aaf386 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:16.044480 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6c2e33-feb9-4930-8ad7-8adb04aaf386 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:16.044480 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6c2e33-feb9-4930-8ad7-8adb04aaf386 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:16.046256 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6c2e33-feb9-4930-8ad7-8adb04aaf386 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:16.053875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 997/1984] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:06:15 2022] PUT /identity/v3/projects/c968576dfbe64995bae40694095cb817/users/8d8e9735e1844b4382b8d797caae6e8c/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:16.058389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68b888b-d611-4e2f-bd54-6e06e00f4673 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:16.060526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 988/1985] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:16 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:06:16.066791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6c2e33-feb9-4930-8ad7-8adb04aaf386 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:16.092010 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6c2e33-feb9-4930-8ad7-8adb04aaf386 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:16.177417 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68b888b-d611-4e2f-bd54-6e06e00f4673 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:16.181842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77189e9d-1262-4bbb-8ac3-cac6105617a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:16.182490 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77189e9d-1262-4bbb-8ac3-cac6105617a0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:16.182803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77189e9d-1262-4bbb-8ac3-cac6105617a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:16.183081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77189e9d-1262-4bbb-8ac3-cac6105617a0 admin admin] PATH_INFO: `/v3/projects/3bc2a6c837a3432b93fd02aefaf735dc/users/b75a423b12b648ea8b340c3a9b6c0245/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:16.184239 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b2ebf9-586a-41d9-9133-59604367270f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:16.185145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b2ebf9-586a-41d9-9133-59604367270f admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:16.185629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b2ebf9-586a-41d9-9133-59604367270f admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:16.185985 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b2ebf9-586a-41d9-9133-59604367270f admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:16.186743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b2ebf9-586a-41d9-9133-59604367270f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:16.188626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b2ebf9-586a-41d9-9133-59604367270f admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:16.201437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77189e9d-1262-4bbb-8ac3-cac6105617a0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3bc2a6c837a3432b93fd02aefaf735dc, user_id=b75a423b12b648ea8b340c3a9b6c0245, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:16.205551 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77189e9d-1262-4bbb-8ac3-cac6105617a0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:16.206292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 998/1986] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:16 2022] GET /identity/v3/roles => generated 1420 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:06:16.207514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b2ebf9-586a-41d9-9133-59604367270f admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:16.220646 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b2ebf9-586a-41d9-9133-59604367270f admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:16.226601 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cfbe7cf-057d-41f8-8326-d8fdb4864cd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:16.227539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfbe7cf-057d-41f8-8326-d8fdb4864cd2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:16.227818 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfbe7cf-057d-41f8-8326-d8fdb4864cd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:16.228092 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cfbe7cf-057d-41f8-8326-d8fdb4864cd2 admin admin] PATH_INFO: `/v3/projects/c968576dfbe64995bae40694095cb817/users/8d8e9735e1844b4382b8d797caae6e8c/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:16.230155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfbe7cf-057d-41f8-8326-d8fdb4864cd2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c968576dfbe64995bae40694095cb817, user_id=8d8e9735e1844b4382b8d797caae6e8c, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:16.232859 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cfbe7cf-057d-41f8-8326-d8fdb4864cd2 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:16.246135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 999/1987] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:06:16 2022] PUT /identity/v3/projects/c968576dfbe64995bae40694095cb817/users/8d8e9735e1844b4382b8d797caae6e8c/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:16.258667 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282eff27-7096-4221-93be-9e9ea283139c None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:16.258952 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282eff27-7096-4221-93be-9e9ea283139c None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:16.259227 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282eff27-7096-4221-93be-9e9ea283139c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:16.290344 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-282eff27-7096-4221-93be-9e9ea283139c None None] MFA Rules not processed for user `8d8e9735e1844b4382b8d797caae6e8c`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:16.293070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:16.293772 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-282eff27-7096-4221-93be-9e9ea283139c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:16.357775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 989/1988] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:06:16 2022] PUT /identity/v3/projects/3bc2a6c837a3432b93fd02aefaf735dc/users/b75a423b12b648ea8b340c3a9b6c0245/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:16.359738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3b163f-fa86-42ab-94c8-4f77621bba36 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:16.359738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3b163f-fa86-42ab-94c8-4f77621bba36 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:16.360135 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3b163f-fa86-42ab-94c8-4f77621bba36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:16.368043 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:16.402428 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-ba3b163f-fa86-42ab-94c8-4f77621bba36 None None] MFA Rules not processed for user `b75a423b12b648ea8b340c3a9b6c0245`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:16.404787 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ba3b163f-fa86-42ab-94c8-4f77621bba36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:16.433921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1000/1989] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:16 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:06:16.446786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282eff27-7096-4221-93be-9e9ea283139c None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:16.449507 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-282eff27-7096-4221-93be-9e9ea283139c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:16.527050 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282eff27-7096-4221-93be-9e9ea283139c None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:16.554494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 990/1990] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:16 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:16.563406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3b163f-fa86-42ab-94c8-4f77621bba36 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:16.582099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:16.591015 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09101237-1a6f-4b6d-bfe2-77322d6c6a9c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:16.593278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09101237-1a6f-4b6d-bfe2-77322d6c6a9c service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:16.593789 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09101237-1a6f-4b6d-bfe2-77322d6c6a9c service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:16.594514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09101237-1a6f-4b6d-bfe2-77322d6c6a9c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:16.596867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-09101237-1a6f-4b6d-bfe2-77322d6c6a9c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:16.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3b163f-fa86-42ab-94c8-4f77621bba36 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:16.657167 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dd1757-7c95-4853-8f47-f6c42bcd2d39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:16.658081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dd1757-7c95-4853-8f47-f6c42bcd2d39 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:16.658367 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dd1757-7c95-4853-8f47-f6c42bcd2d39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:16.658640 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dd1757-7c95-4853-8f47-f6c42bcd2d39 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:16.659481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47dd1757-7c95-4853-8f47-f6c42bcd2d39 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:16.661678 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47dd1757-7c95-4853-8f47-f6c42bcd2d39 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:16.667989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:16.671508 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09101237-1a6f-4b6d-bfe2-77322d6c6a9c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:16.673028 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09101237-1a6f-4b6d-bfe2-77322d6c6a9c service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:16.682999 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1001/1991] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:16 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 237 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:06:16.757242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:16.767160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 991/1992] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:06:16 2022] POST /identity/v3/projects => generated 385 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:06:16.768965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09101237-1a6f-4b6d-bfe2-77322d6c6a9c service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:16.782521 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09101237-1a6f-4b6d-bfe2-77322d6c6a9c service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:16.866198 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5049c1a3-5cf4-4fee-bac8-3e63fb750ac0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:16.867087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5049c1a3-5cf4-4fee-bac8-3e63fb750ac0 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:16.867374 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5049c1a3-5cf4-4fee-bac8-3e63fb750ac0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:16.867644 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5049c1a3-5cf4-4fee-bac8-3e63fb750ac0 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:16.868945 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5049c1a3-5cf4-4fee-bac8-3e63fb750ac0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:16.870951 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5049c1a3-5cf4-4fee-bac8-3e63fb750ac0 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:16.979112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1002/1993] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:06:16 2022] POST /identity/v3/users => generated 293 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:06:16.983024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dd1757-7c95-4853-8f47-f6c42bcd2d39 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:16.993667 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dd1757-7c95-4853-8f47-f6c42bcd2d39 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:16.998268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f90b37-4146-47ba-b280-12a7e648f370 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:16.998859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f90b37-4146-47ba-b280-12a7e648f370 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:16.999136 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f90b37-4146-47ba-b280-12a7e648f370 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:16.999405 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f90b37-4146-47ba-b280-12a7e648f370 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:17.000129 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f90b37-4146-47ba-b280-12a7e648f370 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:17.002159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f90b37-4146-47ba-b280-12a7e648f370 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:17.012959 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 992/1994] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:16 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:06:17.017062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5049c1a3-5cf4-4fee-bac8-3e63fb750ac0 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:17.029177 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5049c1a3-5cf4-4fee-bac8-3e63fb750ac0 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:17.033766 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f48f038-ecfc-41c6-a313-8c3ccc3e0ac0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:17.035241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f48f038-ecfc-41c6-a313-8c3ccc3e0ac0 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:17.035730 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f48f038-ecfc-41c6-a313-8c3ccc3e0ac0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:17.036018 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f48f038-ecfc-41c6-a313-8c3ccc3e0ac0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:17.036674 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f48f038-ecfc-41c6-a313-8c3ccc3e0ac0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:17.038483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f48f038-ecfc-41c6-a313-8c3ccc3e0ac0 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:17.049152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1003/1995] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:17 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:06:17.053385 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f90b37-4146-47ba-b280-12a7e648f370 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:17.063555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f90b37-4146-47ba-b280-12a7e648f370 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:17.069217 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd5fa17-4a19-4a01-906c-30edab90bd7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:17.069897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd5fa17-4a19-4a01-906c-30edab90bd7e admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:17.070173 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd5fa17-4a19-4a01-906c-30edab90bd7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:17.070442 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd5fa17-4a19-4a01-906c-30edab90bd7e admin admin] PATH_INFO: `/v3/projects/746196b139864c45b9917cca6e7c8f79/users/940bbaa778d544fead55a5e3d026f2f6/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:17.086276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd5fa17-4a19-4a01-906c-30edab90bd7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=746196b139864c45b9917cca6e7c8f79, user_id=940bbaa778d544fead55a5e3d026f2f6, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:17.088930 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd5fa17-4a19-4a01-906c-30edab90bd7e admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:17.155631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 993/1996] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:06:17 2022] PUT /identity/v3/projects/746196b139864c45b9917cca6e7c8f79/users/940bbaa778d544fead55a5e3d026f2f6/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:17.166529 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7ca80d-a84f-4e63-b4ba-b8b685e11d39 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:17.166814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7ca80d-a84f-4e63-b4ba-b8b685e11d39 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:17.167092 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7ca80d-a84f-4e63-b4ba-b8b685e11d39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:17.199179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-8d7ca80d-a84f-4e63-b4ba-b8b685e11d39 None None] MFA Rules not processed for user `940bbaa778d544fead55a5e3d026f2f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:17.200404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-8d7ca80d-a84f-4e63-b4ba-b8b685e11d39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:17.269322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1004/1997] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:17 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:17.273976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd5fa17-4a19-4a01-906c-30edab90bd7e admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:17.309280 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:17.331850 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:17.345663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd5fa17-4a19-4a01-906c-30edab90bd7e admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:17.351378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45536265-f34e-4e3e-b890-d68e8e33a679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:17.352091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45536265-f34e-4e3e-b890-d68e8e33a679 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:17.352448 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45536265-f34e-4e3e-b890-d68e8e33a679 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:17.352791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45536265-f34e-4e3e-b890-d68e8e33a679 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:17.353540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45536265-f34e-4e3e-b890-d68e8e33a679 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:17.355679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45536265-f34e-4e3e-b890-d68e8e33a679 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:17.517469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 994/1998] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:06:17 2022] POST /identity/v3/projects => generated 385 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:06:17.518794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7ca80d-a84f-4e63-b4ba-b8b685e11d39 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:17.545287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7ca80d-a84f-4e63-b4ba-b8b685e11d39 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:17.631339 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0921db95-6b38-4867-85dc-ab4c10280cdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:17.632071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0921db95-6b38-4867-85dc-ab4c10280cdc admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:17.632440 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0921db95-6b38-4867-85dc-ab4c10280cdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:17.632861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0921db95-6b38-4867-85dc-ab4c10280cdc admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:17.633708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0921db95-6b38-4867-85dc-ab4c10280cdc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:17.644280 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0921db95-6b38-4867-85dc-ab4c10280cdc admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:17.768349 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1005/1999] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:06:17 2022] POST /identity/v3/users => generated 299 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:06:17.772406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45536265-f34e-4e3e-b890-d68e8e33a679 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:17.786676 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45536265-f34e-4e3e-b890-d68e8e33a679 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:17.792170 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b2a917-411c-41a3-b717-7891467dde42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:17.797267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b2a917-411c-41a3-b717-7891467dde42 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:17.797267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b2a917-411c-41a3-b717-7891467dde42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:17.797267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b2a917-411c-41a3-b717-7891467dde42 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:17.797267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b2a917-411c-41a3-b717-7891467dde42 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:17.797267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b2a917-411c-41a3-b717-7891467dde42 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:17.819727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 995/2000] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:17 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:06:17.823490 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0921db95-6b38-4867-85dc-ab4c10280cdc admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:17.839194 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0921db95-6b38-4867-85dc-ab4c10280cdc admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:17.847281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8627ba1-4d5f-4dcc-8950-71f2784d4f89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:17.847281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8627ba1-4d5f-4dcc-8950-71f2784d4f89 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:17.847281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8627ba1-4d5f-4dcc-8950-71f2784d4f89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:17.847281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8627ba1-4d5f-4dcc-8950-71f2784d4f89 admin admin] PATH_INFO: `/v3/projects/55a945e6448f45deac4fbce8d4e5cdba/users/edb6910b41934b308987502b79e088e2/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:17.865538 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8627ba1-4d5f-4dcc-8950-71f2784d4f89 admin admin] RBAC: Authorizing `identity:create_grant(project_id=55a945e6448f45deac4fbce8d4e5cdba, user_id=edb6910b41934b308987502b79e088e2, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:17.868912 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8627ba1-4d5f-4dcc-8950-71f2784d4f89 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:17.968013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1006/2001] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:06:17 2022] PUT /identity/v3/projects/55a945e6448f45deac4fbce8d4e5cdba/users/edb6910b41934b308987502b79e088e2/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:17.972969 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b2a917-411c-41a3-b717-7891467dde42 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:17.985066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b2a917-411c-41a3-b717-7891467dde42 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:18.059995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4e8492-fbae-478f-9a85-ec6cbea531a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:18.061367 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4e8492-fbae-478f-9a85-ec6cbea531a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:18.061830 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4e8492-fbae-478f-9a85-ec6cbea531a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:18.062435 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4e8492-fbae-478f-9a85-ec6cbea531a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:18.063897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4e8492-fbae-478f-9a85-ec6cbea531a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:18.070412 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:18.070817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4e8492-fbae-478f-9a85-ec6cbea531a8 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:18.077720 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 996/2002] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:17 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:06:18.081910 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8627ba1-4d5f-4dcc-8950-71f2784d4f89 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:18.095373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8627ba1-4d5f-4dcc-8950-71f2784d4f89 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:18.101130 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c3d69a-6e19-4940-9f79-a2284cca04b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:18.101904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c3d69a-6e19-4940-9f79-a2284cca04b5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:18.102264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c3d69a-6e19-4940-9f79-a2284cca04b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:18.102615 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c3d69a-6e19-4940-9f79-a2284cca04b5 admin admin] PATH_INFO: `/v3/projects/55a945e6448f45deac4fbce8d4e5cdba/users/edb6910b41934b308987502b79e088e2/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:18.104754 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c3d69a-6e19-4940-9f79-a2284cca04b5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=55a945e6448f45deac4fbce8d4e5cdba, user_id=edb6910b41934b308987502b79e088e2, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:18.107622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c3d69a-6e19-4940-9f79-a2284cca04b5 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:18.124390 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1007/2003] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:06:18 2022] PUT /identity/v3/projects/55a945e6448f45deac4fbce8d4e5cdba/users/edb6910b41934b308987502b79e088e2/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:18.136552 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce4bf62-3a83-4f45-8f0e-24d25c304bf1 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:18.138218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce4bf62-3a83-4f45-8f0e-24d25c304bf1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:18.141220 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce4bf62-3a83-4f45-8f0e-24d25c304bf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:18.169181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-0ce4bf62-3a83-4f45-8f0e-24d25c304bf1 None None] MFA Rules not processed for user `edb6910b41934b308987502b79e088e2`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:18.171888 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-0ce4bf62-3a83-4f45-8f0e-24d25c304bf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:18.253329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 997/2004] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:18 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:18.266237 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c3d69a-6e19-4940-9f79-a2284cca04b5 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:18.267990 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-10c3d69a-6e19-4940-9f79-a2284cca04b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:18.327081 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c3d69a-6e19-4940-9f79-a2284cca04b5 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:18.330949 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8feaef42-10bc-4f2d-9e47-c57783c02437 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:18.331554 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feaef42-10bc-4f2d-9e47-c57783c02437 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:18.331831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feaef42-10bc-4f2d-9e47-c57783c02437 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:18.332134 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8feaef42-10bc-4f2d-9e47-c57783c02437 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:18.365769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:18.396992 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8feaef42-10bc-4f2d-9e47-c57783c02437 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:18.398704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8feaef42-10bc-4f2d-9e47-c57783c02437 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:18.409811 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1008/2005] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:18 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:18.423031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:20.692614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce4bf62-3a83-4f45-8f0e-24d25c304bf1 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:20.702301 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce4bf62-3a83-4f45-8f0e-24d25c304bf1 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:20.706732 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996867c3-8289-45bd-8f0f-a0a8fae8c75f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:20.707325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996867c3-8289-45bd-8f0f-a0a8fae8c75f service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:20.707603 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996867c3-8289-45bd-8f0f-a0a8fae8c75f service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:20.707873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996867c3-8289-45bd-8f0f-a0a8fae8c75f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:20.732952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:20.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996867c3-8289-45bd-8f0f-a0a8fae8c75f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:20.767084 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996867c3-8289-45bd-8f0f-a0a8fae8c75f service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:20.776476 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 998/2006] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:20 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:22.605685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8feaef42-10bc-4f2d-9e47-c57783c02437 service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:22.614828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8feaef42-10bc-4f2d-9e47-c57783c02437 service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:22.618547 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c45c99-428e-4a56-af51-0d515de3e9b6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:22.619151 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c45c99-428e-4a56-af51-0d515de3e9b6 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:22.619430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c45c99-428e-4a56-af51-0d515de3e9b6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:22.619702 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c45c99-428e-4a56-af51-0d515de3e9b6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:22.629608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c45c99-428e-4a56-af51-0d515de3e9b6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:22.631059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c45c99-428e-4a56-af51-0d515de3e9b6 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:22.641337 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1009/2007] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:22 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:23.605761 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996867c3-8289-45bd-8f0f-a0a8fae8c75f service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:23.618858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996867c3-8289-45bd-8f0f-a0a8fae8c75f service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:23.622869 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166e5a24-30fa-4304-8712-c322669af8fb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:23.623579 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166e5a24-30fa-4304-8712-c322669af8fb service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:23.623860 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166e5a24-30fa-4304-8712-c322669af8fb service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:23.624408 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166e5a24-30fa-4304-8712-c322669af8fb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:23.695230 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166e5a24-30fa-4304-8712-c322669af8fb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:23.705183 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166e5a24-30fa-4304-8712-c322669af8fb service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:23.714869 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 999/2008] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:23 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:28.506452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c45c99-428e-4a56-af51-0d515de3e9b6 service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:28.583298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c45c99-428e-4a56-af51-0d515de3e9b6 service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:28.588050 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0598efdb-0098-4400-a936-29a738fd2269 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:28.592357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0598efdb-0098-4400-a936-29a738fd2269 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:28.592357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0598efdb-0098-4400-a936-29a738fd2269 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:28.592357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0598efdb-0098-4400-a936-29a738fd2269 admin admin] PATH_INFO: `/v3/users/b75a423b12b648ea8b340c3a9b6c0245` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:28.592357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0598efdb-0098-4400-a936-29a738fd2269 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b75a423b12b648ea8b340c3a9b6c0245)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:28.592357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0598efdb-0098-4400-a936-29a738fd2269 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:28.826635 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0598efdb-0098-4400-a936-29a738fd2269 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b75a423b12b648ea8b340c3a9b6c0245'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:28.920738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0598efdb-0098-4400-a936-29a738fd2269 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b75a423b12b648ea8b340c3a9b6c0245'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:28.934789 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0598efdb-0098-4400-a936-29a738fd2269 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b75a423b12b648ea8b340c3a9b6c0245'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:28.935681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0598efdb-0098-4400-a936-29a738fd2269 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b75a423b12b648ea8b340c3a9b6c0245'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:28.942830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0598efdb-0098-4400-a936-29a738fd2269 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b75a423b12b648ea8b340c3a9b6c0245'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:28.949486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1010/2009] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:28 2022] DELETE /identity/v3/users/b75a423b12b648ea8b340c3a9b6c0245 => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:28.954061 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283802e4-0ecf-4ca7-8e64-73820275c383 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:28.954546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283802e4-0ecf-4ca7-8e64-73820275c383 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:28.955929 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283802e4-0ecf-4ca7-8e64-73820275c383 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:28.978483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-283802e4-0ecf-4ca7-8e64-73820275c383 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:28.979917 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-283802e4-0ecf-4ca7-8e64-73820275c383 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:29.046981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1000/2010] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:29.061735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0598efdb-0098-4400-a936-29a738fd2269 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:29.063226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0598efdb-0098-4400-a936-29a738fd2269 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:29.105112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:29.127756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0598efdb-0098-4400-a936-29a738fd2269 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:29.131412 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13a01a8-8b5f-4d37-9625-9e430145e704 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:29.132046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13a01a8-8b5f-4d37-9625-9e430145e704 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:29.132320 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13a01a8-8b5f-4d37-9625-9e430145e704 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:29.132590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13a01a8-8b5f-4d37-9625-9e430145e704 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:29.144106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13a01a8-8b5f-4d37-9625-9e430145e704 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:29.150086 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13a01a8-8b5f-4d37-9625-9e430145e704 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:29.156695 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1011/2011] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:29.644612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283802e4-0ecf-4ca7-8e64-73820275c383 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:29.647736 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-283802e4-0ecf-4ca7-8e64-73820275c383 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:29.677242 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283802e4-0ecf-4ca7-8e64-73820275c383 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:29.683058 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3650694-2a8f-4055-914a-16384f6b3930 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:29.683828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3650694-2a8f-4055-914a-16384f6b3930 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:29.684174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3650694-2a8f-4055-914a-16384f6b3930 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:29.684518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3650694-2a8f-4055-914a-16384f6b3930 admin admin] PATH_INFO: `/v3/projects/3bc2a6c837a3432b93fd02aefaf735dc` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:29.686070 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3650694-2a8f-4055-914a-16384f6b3930 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3bc2a6c837a3432b93fd02aefaf735dc)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:29.688383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3650694-2a8f-4055-914a-16384f6b3930 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:29.779817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a3650694-2a8f-4055-914a-16384f6b3930 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3bc2a6c837a3432b93fd02aefaf735dc', 'request_id': 'req-a3650694-2a8f-4055-914a-16384f6b3930'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:29.784317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:29.786579 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a3650694-2a8f-4055-914a-16384f6b3930 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3bc2a6c837a3432b93fd02aefaf735dc', 'request_id': 'req-a3650694-2a8f-4055-914a-16384f6b3930'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:29.847256 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13a01a8-8b5f-4d37-9625-9e430145e704 service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:29.847780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a3650694-2a8f-4055-914a-16384f6b3930 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3bc2a6c837a3432b93fd02aefaf735dc', 'request_id': 'req-a3650694-2a8f-4055-914a-16384f6b3930'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:29.848747 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b13a01a8-8b5f-4d37-9625-9e430145e704 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:29.864068 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a3650694-2a8f-4055-914a-16384f6b3930 admin admin] The token cache is being invalidate because project 3bc2a6c837a3432b93fd02aefaf735dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:29.864701 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a3650694-2a8f-4055-914a-16384f6b3930 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:29.865264 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a3650694-2a8f-4055-914a-16384f6b3930 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:29.866018 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13a01a8-8b5f-4d37-9625-9e430145e704 service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:29.870179 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1001/2012] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:29 2022] DELETE /identity/v3/projects/3bc2a6c837a3432b93fd02aefaf735dc => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:29.871164 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3650694-2a8f-4055-914a-16384f6b3930 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:29.872643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a3650694-2a8f-4055-914a-16384f6b3930 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:29.965748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1253464-8df2-455f-8778-305a7ca39805 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:29.966553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1253464-8df2-455f-8778-305a7ca39805 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:29.966925 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1253464-8df2-455f-8778-305a7ca39805 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:29.967301 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1253464-8df2-455f-8778-305a7ca39805 admin admin] PATH_INFO: `/v3/users/47fb156027d14eb6a014bc9b9c89e9f6` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:29.968820 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1253464-8df2-455f-8778-305a7ca39805 admin admin] RBAC: Authorizing `identity:delete_user(user_id=47fb156027d14eb6a014bc9b9c89e9f6)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:29.971596 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1253464-8df2-455f-8778-305a7ca39805 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:29.973448 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3650694-2a8f-4055-914a-16384f6b3930 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:29.978944 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b327697-80e7-408c-aa0e-39e0c8efe37f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:29.979597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b327697-80e7-408c-aa0e-39e0c8efe37f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:29.979884 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b327697-80e7-408c-aa0e-39e0c8efe37f admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:29.980160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b327697-80e7-408c-aa0e-39e0c8efe37f admin admin] PATH_INFO: `/v3/users/940bbaa778d544fead55a5e3d026f2f6` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:29.981359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b327697-80e7-408c-aa0e-39e0c8efe37f admin admin] RBAC: Authorizing `identity:delete_user(user_id=940bbaa778d544fead55a5e3d026f2f6)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:29.983308 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b327697-80e7-408c-aa0e-39e0c8efe37f admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:30.163955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b1253464-8df2-455f-8778-305a7ca39805 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '47fb156027d14eb6a014bc9b9c89e9f6'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:30.220994 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-6b327697-80e7-408c-aa0e-39e0c8efe37f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '940bbaa778d544fead55a5e3d026f2f6'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:30.221680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-6b327697-80e7-408c-aa0e-39e0c8efe37f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '940bbaa778d544fead55a5e3d026f2f6'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:30.229496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b1253464-8df2-455f-8778-305a7ca39805 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '47fb156027d14eb6a014bc9b9c89e9f6'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:30.243593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b1253464-8df2-455f-8778-305a7ca39805 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '47fb156027d14eb6a014bc9b9c89e9f6'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:30.244303 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b1253464-8df2-455f-8778-305a7ca39805 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '47fb156027d14eb6a014bc9b9c89e9f6'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:30.251593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b1253464-8df2-455f-8778-305a7ca39805 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '47fb156027d14eb6a014bc9b9c89e9f6'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:30.257859 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1012/2013] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:29 2022] DELETE /identity/v3/users/47fb156027d14eb6a014bc9b9c89e9f6 => generated 0 bytes in 412 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:06:30.269499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bcbda1-c7a5-4e06-a60e-c4321cd70690 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:30.269499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bcbda1-c7a5-4e06-a60e-c4321cd70690 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:30.269499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bcbda1-c7a5-4e06-a60e-c4321cd70690 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:30.281207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-b7bcbda1-c7a5-4e06-a60e-c4321cd70690 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:30.282411 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b7bcbda1-c7a5-4e06-a60e-c4321cd70690 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:30.303786 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-6b327697-80e7-408c-aa0e-39e0c8efe37f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '940bbaa778d544fead55a5e3d026f2f6'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:30.320313 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-6b327697-80e7-408c-aa0e-39e0c8efe37f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '940bbaa778d544fead55a5e3d026f2f6'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:30.321234 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-6b327697-80e7-408c-aa0e-39e0c8efe37f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '940bbaa778d544fead55a5e3d026f2f6'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:30.333849 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:30.338568 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1002/2014] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:29 2022] DELETE /identity/v3/users/940bbaa778d544fead55a5e3d026f2f6 => generated 0 bytes in 468 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:30.348648 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1013/2015] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:06:30.358305 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b327697-80e7-408c-aa0e-39e0c8efe37f admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:30.359736 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6b327697-80e7-408c-aa0e-39e0c8efe37f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:30.423442 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b327697-80e7-408c-aa0e-39e0c8efe37f admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:30.427093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fe9ceb-65a3-4109-9542-3a98b0a89f79 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:30.427712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fe9ceb-65a3-4109-9542-3a98b0a89f79 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:30.427995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fe9ceb-65a3-4109-9542-3a98b0a89f79 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:30.428273 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fe9ceb-65a3-4109-9542-3a98b0a89f79 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:30.443194 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:30.493288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fe9ceb-65a3-4109-9542-3a98b0a89f79 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:30.494758 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fe9ceb-65a3-4109-9542-3a98b0a89f79 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:30.504335 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1003/2016] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:30.866193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bcbda1-c7a5-4e06-a60e-c4321cd70690 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:30.868446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b7bcbda1-c7a5-4e06-a60e-c4321cd70690 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:30.884794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bcbda1-c7a5-4e06-a60e-c4321cd70690 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:30.890941 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eade2cd-f67d-477e-ae39-a8d0f519cb17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:30.891974 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eade2cd-f67d-477e-ae39-a8d0f519cb17 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:30.892401 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eade2cd-f67d-477e-ae39-a8d0f519cb17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:30.892816 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eade2cd-f67d-477e-ae39-a8d0f519cb17 admin admin] PATH_INFO: `/v3/projects/746196b139864c45b9917cca6e7c8f79` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:30.894323 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eade2cd-f67d-477e-ae39-a8d0f519cb17 admin admin] RBAC: Authorizing `identity:delete_project(project_id=746196b139864c45b9917cca6e7c8f79)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:30.897974 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eade2cd-f67d-477e-ae39-a8d0f519cb17 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:31.060781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fe9ceb-65a3-4109-9542-3a98b0a89f79 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:31.062376 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-61fe9ceb-65a3-4109-9542-3a98b0a89f79 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:31.090126 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fe9ceb-65a3-4109-9542-3a98b0a89f79 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:31.092962 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-6eade2cd-f67d-477e-ae39-a8d0f519cb17 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '746196b139864c45b9917cca6e7c8f79', 'request_id': 'req-6eade2cd-f67d-477e-ae39-a8d0f519cb17'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:31.100285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-6eade2cd-f67d-477e-ae39-a8d0f519cb17 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '746196b139864c45b9917cca6e7c8f79', 'request_id': 'req-6eade2cd-f67d-477e-ae39-a8d0f519cb17'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:31.125722 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-6eade2cd-f67d-477e-ae39-a8d0f519cb17 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '746196b139864c45b9917cca6e7c8f79', 'request_id': 'req-6eade2cd-f67d-477e-ae39-a8d0f519cb17'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:31.204395 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f01c6f-80a9-4a3e-a06a-63c333d752d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:31.205521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f01c6f-80a9-4a3e-a06a-63c333d752d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:31.205974 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f01c6f-80a9-4a3e-a06a-63c333d752d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:31.207111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f01c6f-80a9-4a3e-a06a-63c333d752d0 admin admin] PATH_INFO: `/v3/projects/2bea56a7089f4fa189405ee93b0993e5` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:31.209569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f01c6f-80a9-4a3e-a06a-63c333d752d0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2bea56a7089f4fa189405ee93b0993e5)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:31.210581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-6eade2cd-f67d-477e-ae39-a8d0f519cb17 admin admin] The token cache is being invalidate because project 746196b139864c45b9917cca6e7c8f79 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:31.210992 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-6eade2cd-f67d-477e-ae39-a8d0f519cb17 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:31.211759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-6eade2cd-f67d-477e-ae39-a8d0f519cb17 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:31.213037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f01c6f-80a9-4a3e-a06a-63c333d752d0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:31.218930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1014/2017] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:30 2022] DELETE /identity/v3/projects/746196b139864c45b9917cca6e7c8f79 => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:31.220877 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eade2cd-f67d-477e-ae39-a8d0f519cb17 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:31.223963 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-6eade2cd-f67d-477e-ae39-a8d0f519cb17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:31.243661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eade2cd-f67d-477e-ae39-a8d0f519cb17 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:31.249712 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1687c3-f16e-4e12-83e9-d69b81ee1fee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:31.250381 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1687c3-f16e-4e12-83e9-d69b81ee1fee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:31.250665 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1687c3-f16e-4e12-83e9-d69b81ee1fee admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:31.250950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1687c3-f16e-4e12-83e9-d69b81ee1fee admin admin] PATH_INFO: `/v3/users/edb6910b41934b308987502b79e088e2` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:31.252130 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1687c3-f16e-4e12-83e9-d69b81ee1fee admin admin] RBAC: Authorizing `identity:delete_user(user_id=edb6910b41934b308987502b79e088e2)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:31.255493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1687c3-f16e-4e12-83e9-d69b81ee1fee admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:31.404144 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-49f01c6f-80a9-4a3e-a06a-63c333d752d0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2bea56a7089f4fa189405ee93b0993e5', 'request_id': 'req-49f01c6f-80a9-4a3e-a06a-63c333d752d0'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:31.411259 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-49f01c6f-80a9-4a3e-a06a-63c333d752d0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2bea56a7089f4fa189405ee93b0993e5', 'request_id': 'req-49f01c6f-80a9-4a3e-a06a-63c333d752d0'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:31.484546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-db1687c3-f16e-4e12-83e9-d69b81ee1fee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'edb6910b41934b308987502b79e088e2'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:31.532091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-49f01c6f-80a9-4a3e-a06a-63c333d752d0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2bea56a7089f4fa189405ee93b0993e5', 'request_id': 'req-49f01c6f-80a9-4a3e-a06a-63c333d752d0'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:31.544470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-49f01c6f-80a9-4a3e-a06a-63c333d752d0 admin admin] The token cache is being invalidate because project 2bea56a7089f4fa189405ee93b0993e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:31.544470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-49f01c6f-80a9-4a3e-a06a-63c333d752d0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:31.544917 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-49f01c6f-80a9-4a3e-a06a-63c333d752d0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:31.546748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1004/2018] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:31 2022] DELETE /identity/v3/projects/2bea56a7089f4fa189405ee93b0993e5 => generated 0 bytes in 486 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:06:31.550559 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f01c6f-80a9-4a3e-a06a-63c333d752d0 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:31.552275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-49f01c6f-80a9-4a3e-a06a-63c333d752d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:31.589211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-db1687c3-f16e-4e12-83e9-d69b81ee1fee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'edb6910b41934b308987502b79e088e2'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:31.603761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-db1687c3-f16e-4e12-83e9-d69b81ee1fee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'edb6910b41934b308987502b79e088e2'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:31.603949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:31.604578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-db1687c3-f16e-4e12-83e9-d69b81ee1fee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'edb6910b41934b308987502b79e088e2'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:31.612400 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-db1687c3-f16e-4e12-83e9-d69b81ee1fee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'edb6910b41934b308987502b79e088e2'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:31.622169 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1015/2019] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:31 2022] DELETE /identity/v3/users/edb6910b41934b308987502b79e088e2 => generated 0 bytes in 403 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:31.624600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f01c6f-80a9-4a3e-a06a-63c333d752d0 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:31.686639 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d74cff2-c53e-4073-ace5-62302b8e3b6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:31.687716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d74cff2-c53e-4073-ace5-62302b8e3b6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:31.688212 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d74cff2-c53e-4073-ace5-62302b8e3b6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:31.689216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d74cff2-c53e-4073-ace5-62302b8e3b6a admin admin] PATH_INFO: `/v3/users/8d8e9735e1844b4382b8d797caae6e8c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:31.690565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d74cff2-c53e-4073-ace5-62302b8e3b6a admin admin] RBAC: Authorizing `identity:delete_user(user_id=8d8e9735e1844b4382b8d797caae6e8c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:31.693197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d74cff2-c53e-4073-ace5-62302b8e3b6a admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:31.911586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-1d74cff2-c53e-4073-ace5-62302b8e3b6a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8d8e9735e1844b4382b8d797caae6e8c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:31.912357 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-1d74cff2-c53e-4073-ace5-62302b8e3b6a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8d8e9735e1844b4382b8d797caae6e8c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:32.016811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-1d74cff2-c53e-4073-ace5-62302b8e3b6a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8d8e9735e1844b4382b8d797caae6e8c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:32.030623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-1d74cff2-c53e-4073-ace5-62302b8e3b6a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8d8e9735e1844b4382b8d797caae6e8c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:32.031582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-1d74cff2-c53e-4073-ace5-62302b8e3b6a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8d8e9735e1844b4382b8d797caae6e8c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:32.043913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1005/2020] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:31 2022] DELETE /identity/v3/users/8d8e9735e1844b4382b8d797caae6e8c => generated 0 bytes in 494 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:06:32.100459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1687c3-f16e-4e12-83e9-d69b81ee1fee admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:32.102717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-db1687c3-f16e-4e12-83e9-d69b81ee1fee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:32.117753 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1687c3-f16e-4e12-83e9-d69b81ee1fee admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:32.166664 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:32.178599 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798d0210-f002-4786-b404-239241c3e617 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:32.179254 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798d0210-f002-4786-b404-239241c3e617 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:32.179545 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798d0210-f002-4786-b404-239241c3e617 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:32.179812 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798d0210-f002-4786-b404-239241c3e617 admin admin] PATH_INFO: `/v3/projects/55a945e6448f45deac4fbce8d4e5cdba` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:32.180968 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798d0210-f002-4786-b404-239241c3e617 admin admin] RBAC: Authorizing `identity:delete_project(project_id=55a945e6448f45deac4fbce8d4e5cdba)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:32.183050 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798d0210-f002-4786-b404-239241c3e617 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:32.332689 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-798d0210-f002-4786-b404-239241c3e617 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '55a945e6448f45deac4fbce8d4e5cdba', 'request_id': 'req-798d0210-f002-4786-b404-239241c3e617'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:32.338109 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-798d0210-f002-4786-b404-239241c3e617 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '55a945e6448f45deac4fbce8d4e5cdba', 'request_id': 'req-798d0210-f002-4786-b404-239241c3e617'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:32.344273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-798d0210-f002-4786-b404-239241c3e617 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '55a945e6448f45deac4fbce8d4e5cdba', 'request_id': 'req-798d0210-f002-4786-b404-239241c3e617'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:32.405890 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-798d0210-f002-4786-b404-239241c3e617 admin admin] The token cache is being invalidate because project 55a945e6448f45deac4fbce8d4e5cdba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:32.406258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-798d0210-f002-4786-b404-239241c3e617 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:32.406899 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-798d0210-f002-4786-b404-239241c3e617 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:32.414437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1016/2021] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:32 2022] DELETE /identity/v3/projects/55a945e6448f45deac4fbce8d4e5cdba => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:32.414701 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f1fea1-9460-4744-9447-4f36789df21c None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:32.415601 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f1fea1-9460-4744-9447-4f36789df21c None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:32.417032 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f1fea1-9460-4744-9447-4f36789df21c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:32.443555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-e9f1fea1-9460-4744-9447-4f36789df21c None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:32.444810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e9f1fea1-9460-4744-9447-4f36789df21c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:32.510005 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1006/2022] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:06:32.517885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efe051b-3084-4e38-91da-5348cdefdc0e None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:32.518370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efe051b-3084-4e38-91da-5348cdefdc0e None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:32.519451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efe051b-3084-4e38-91da-5348cdefdc0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:32.541518 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9efe051b-3084-4e38-91da-5348cdefdc0e None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:32.542755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9efe051b-3084-4e38-91da-5348cdefdc0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:32.553256 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1017/2023] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:32.556171 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f1fea1-9460-4744-9447-4f36789df21c None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:32.566767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f1fea1-9460-4744-9447-4f36789df21c None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:32.573398 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbeef9f1-785d-481c-bafe-298f3ac58cf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:32.573398 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbeef9f1-785d-481c-bafe-298f3ac58cf5 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:32.573578 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbeef9f1-785d-481c-bafe-298f3ac58cf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:32.574844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbeef9f1-785d-481c-bafe-298f3ac58cf5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:32.577759 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbeef9f1-785d-481c-bafe-298f3ac58cf5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:06:32.578081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbeef9f1-785d-481c-bafe-298f3ac58cf5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:32.582233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbeef9f1-785d-481c-bafe-298f3ac58cf5 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:32.583233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:32.593121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1007/2024] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:06:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 01:06:32.597405 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9efe051b-3084-4e38-91da-5348cdefdc0e None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:32.609004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9efe051b-3084-4e38-91da-5348cdefdc0e None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:32.614062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4338c330-ec70-4f98-9f91-4527597c1d24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:32.615071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4338c330-ec70-4f98-9f91-4527597c1d24 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:32.615418 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4338c330-ec70-4f98-9f91-4527597c1d24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:32.615757 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4338c330-ec70-4f98-9f91-4527597c1d24 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:32.616473 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4338c330-ec70-4f98-9f91-4527597c1d24 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:32.618484 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4338c330-ec70-4f98-9f91-4527597c1d24 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:32.662562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbeef9f1-785d-481c-bafe-298f3ac58cf5 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:32.664319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-cbeef9f1-785d-481c-bafe-298f3ac58cf5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:32.678250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbeef9f1-785d-481c-bafe-298f3ac58cf5 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:32.684218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fc76b1-169c-4fe3-a5aa-31c836122f40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:32.684864 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fc76b1-169c-4fe3-a5aa-31c836122f40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:32.685374 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fc76b1-169c-4fe3-a5aa-31c836122f40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:32.685657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fc76b1-169c-4fe3-a5aa-31c836122f40 admin admin] PATH_INFO: `/v3/projects/c968576dfbe64995bae40694095cb817` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:32.686982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fc76b1-169c-4fe3-a5aa-31c836122f40 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c968576dfbe64995bae40694095cb817)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:32.688949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fc76b1-169c-4fe3-a5aa-31c836122f40 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:32.716279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1018/2025] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:06:32 2022] POST /identity/v3/projects => generated 383 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:06:32.717497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4338c330-ec70-4f98-9f91-4527597c1d24 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:32.785446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4338c330-ec70-4f98-9f91-4527597c1d24 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:32.795475 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d5fc76b1-169c-4fe3-a5aa-31c836122f40 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c968576dfbe64995bae40694095cb817', 'request_id': 'req-d5fc76b1-169c-4fe3-a5aa-31c836122f40'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:32.802557 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d5fc76b1-169c-4fe3-a5aa-31c836122f40 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c968576dfbe64995bae40694095cb817', 'request_id': 'req-d5fc76b1-169c-4fe3-a5aa-31c836122f40'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:32.848808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e80449-fb71-496a-8e22-8ee357d6ee49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:32.849486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e80449-fb71-496a-8e22-8ee357d6ee49 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:32.849770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e80449-fb71-496a-8e22-8ee357d6ee49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:32.850045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e80449-fb71-496a-8e22-8ee357d6ee49 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:32.850688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e80449-fb71-496a-8e22-8ee357d6ee49 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:32.852650 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e80449-fb71-496a-8e22-8ee357d6ee49 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:32.860892 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:32.886486 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d5fc76b1-169c-4fe3-a5aa-31c836122f40 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c968576dfbe64995bae40694095cb817', 'request_id': 'req-d5fc76b1-169c-4fe3-a5aa-31c836122f40'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:32.900444 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d5fc76b1-169c-4fe3-a5aa-31c836122f40 admin admin] The token cache is being invalidate because project c968576dfbe64995bae40694095cb817 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:32.900738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d5fc76b1-169c-4fe3-a5aa-31c836122f40 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:32.901451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d5fc76b1-169c-4fe3-a5aa-31c836122f40 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:32.904255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1008/2026] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:32 2022] DELETE /identity/v3/projects/c968576dfbe64995bae40694095cb817 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:06:32.909057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62a516c-5086-4a75-ad82-78b263c68d36 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:32.914211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62a516c-5086-4a75-ad82-78b263c68d36 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:32.914211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62a516c-5086-4a75-ad82-78b263c68d36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:32.932359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-f62a516c-5086-4a75-ad82-78b263c68d36 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:32.933671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f62a516c-5086-4a75-ad82-78b263c68d36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:32.942276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1009/2027] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:06:32.950948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b8e61a-691a-4f78-9188-ce0a0ca1d886 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:32.951242 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b8e61a-691a-4f78-9188-ce0a0ca1d886 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:32.951512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b8e61a-691a-4f78-9188-ce0a0ca1d886 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:32.970808 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-49b8e61a-691a-4f78-9188-ce0a0ca1d886 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:32.972037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-49b8e61a-691a-4f78-9188-ce0a0ca1d886 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:32.977317 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1019/2028] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:06:32 2022] POST /identity/v3/users => generated 292 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:06:32.980186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e80449-fb71-496a-8e22-8ee357d6ee49 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:32.983166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-00e80449-fb71-496a-8e22-8ee357d6ee49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:32.985111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1010/2029] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:06:32.990304 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b8e61a-691a-4f78-9188-ce0a0ca1d886 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:33.000830 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b8e61a-691a-4f78-9188-ce0a0ca1d886 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:33.006328 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e80449-fb71-496a-8e22-8ee357d6ee49 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:33.006994 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac41a699-42f0-49c0-b94f-08744a30f216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:33.007611 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac41a699-42f0-49c0-b94f-08744a30f216 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:33.007882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac41a699-42f0-49c0-b94f-08744a30f216 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:33.008152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac41a699-42f0-49c0-b94f-08744a30f216 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:33.008811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac41a699-42f0-49c0-b94f-08744a30f216 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:06:33.009138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac41a699-42f0-49c0-b94f-08744a30f216 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:33.010773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac41a699-42f0-49c0-b94f-08744a30f216 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:33.012505 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388fda1f-a4f9-4332-8aac-e21b226fea85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:33.013158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388fda1f-a4f9-4332-8aac-e21b226fea85 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:33.013446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388fda1f-a4f9-4332-8aac-e21b226fea85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:33.013723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388fda1f-a4f9-4332-8aac-e21b226fea85 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:33.014381 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388fda1f-a4f9-4332-8aac-e21b226fea85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:33.016476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-388fda1f-a4f9-4332-8aac-e21b226fea85 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:33.023786 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1011/2030] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:06:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 01:06:33.029862 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1020/2031] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:32 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:06:33.030541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac41a699-42f0-49c0-b94f-08744a30f216 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:33.033544 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388fda1f-a4f9-4332-8aac-e21b226fea85 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:33.045761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388fda1f-a4f9-4332-8aac-e21b226fea85 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:33.046564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac41a699-42f0-49c0-b94f-08744a30f216 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:33.051312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fd456d-a8a1-4b37-8a5a-363961bd4dcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:33.052125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fd456d-a8a1-4b37-8a5a-363961bd4dcc admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:33.052474 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fd456d-a8a1-4b37-8a5a-363961bd4dcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:33.052743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fd456d-a8a1-4b37-8a5a-363961bd4dcc admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:33.053428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fd456d-a8a1-4b37-8a5a-363961bd4dcc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:33.054094 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6eac2e-7821-4b80-95d3-b08376a85a97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:33.054708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6eac2e-7821-4b80-95d3-b08376a85a97 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:33.054992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6eac2e-7821-4b80-95d3-b08376a85a97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:33.055103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fd456d-a8a1-4b37-8a5a-363961bd4dcc admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:33.055269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6eac2e-7821-4b80-95d3-b08376a85a97 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:33.055962 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6eac2e-7821-4b80-95d3-b08376a85a97 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:33.057999 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6eac2e-7821-4b80-95d3-b08376a85a97 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:33.071528 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1021/2032] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:33 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:06:33.072571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fd456d-a8a1-4b37-8a5a-363961bd4dcc admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:33.085041 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fd456d-a8a1-4b37-8a5a-363961bd4dcc admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:33.090026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eff5295-b640-4f63-87ef-a351466b29e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:33.090682 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eff5295-b640-4f63-87ef-a351466b29e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:33.090961 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eff5295-b640-4f63-87ef-a351466b29e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:33.091252 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eff5295-b640-4f63-87ef-a351466b29e9 admin admin] PATH_INFO: `/v3/projects/6071c3a795ba4aa098e3af8cde306d58/users/77de7bb68f3c45d680c8a78dc40df07b/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:33.107400 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eff5295-b640-4f63-87ef-a351466b29e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6071c3a795ba4aa098e3af8cde306d58, user_id=77de7bb68f3c45d680c8a78dc40df07b, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:33.110141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eff5295-b640-4f63-87ef-a351466b29e9 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:33.245974 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1012/2033] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:06:33 2022] POST /identity/v3/projects => generated 369 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:06:33.250469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6eac2e-7821-4b80-95d3-b08376a85a97 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:33.274046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1022/2034] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:06:33 2022] PUT /identity/v3/projects/6071c3a795ba4aa098e3af8cde306d58/users/77de7bb68f3c45d680c8a78dc40df07b/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:33.278998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bc70e5-901f-48f7-b19c-69c2376b9d62 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:33.279278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bc70e5-901f-48f7-b19c-69c2376b9d62 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:33.279551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bc70e5-901f-48f7-b19c-69c2376b9d62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:33.299262 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-34bc70e5-901f-48f7-b19c-69c2376b9d62 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:33.300411 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-34bc70e5-901f-48f7-b19c-69c2376b9d62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:33.328749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6eac2e-7821-4b80-95d3-b08376a85a97 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:33.370024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:33.372085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1023/2035] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:06:33.380927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bc70e5-901f-48f7-b19c-69c2376b9d62 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:33.382769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-34bc70e5-901f-48f7-b19c-69c2376b9d62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:33.412103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8137271f-9fed-4253-8c91-052f102e6426 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:33.412734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8137271f-9fed-4253-8c91-052f102e6426 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:33.413018 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8137271f-9fed-4253-8c91-052f102e6426 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:33.413426 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8137271f-9fed-4253-8c91-052f102e6426 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:33.414222 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8137271f-9fed-4253-8c91-052f102e6426 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:33.416647 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8137271f-9fed-4253-8c91-052f102e6426 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:33.468082 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bc70e5-901f-48f7-b19c-69c2376b9d62 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:33.471914 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c0eede-3556-4d5f-aa63-fb2036809735 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:33.472679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c0eede-3556-4d5f-aa63-fb2036809735 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:33.473149 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c0eede-3556-4d5f-aa63-fb2036809735 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:33.473427 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c0eede-3556-4d5f-aa63-fb2036809735 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:33.483485 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c0eede-3556-4d5f-aa63-fb2036809735 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:33.485114 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c0eede-3556-4d5f-aa63-fb2036809735 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:33.498479 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1024/2036] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:33.504137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1013/2037] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:06:33 2022] POST /identity/v3/users => generated 285 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:06:33.504416 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c0eede-3556-4d5f-aa63-fb2036809735 service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:33.513921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c0eede-3556-4d5f-aa63-fb2036809735 service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:33.519867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c24eb4c-e11b-402e-91a5-8e14698e0241 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:33.520547 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c24eb4c-e11b-402e-91a5-8e14698e0241 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:33.520831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c24eb4c-e11b-402e-91a5-8e14698e0241 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:33.521234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c24eb4c-e11b-402e-91a5-8e14698e0241 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:33.521961 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c24eb4c-e11b-402e-91a5-8e14698e0241 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:33.524384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c24eb4c-e11b-402e-91a5-8e14698e0241 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:33.539267 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1025/2038] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:33 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:06:33.543536 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8137271f-9fed-4253-8c91-052f102e6426 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:33.555023 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8137271f-9fed-4253-8c91-052f102e6426 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:33.560084 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ac1678-3623-4080-8982-b6741f79e623 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:33.560819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ac1678-3623-4080-8982-b6741f79e623 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:33.561202 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ac1678-3623-4080-8982-b6741f79e623 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:33.561546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ac1678-3623-4080-8982-b6741f79e623 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:33.562427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ac1678-3623-4080-8982-b6741f79e623 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:33.564100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ac1678-3623-4080-8982-b6741f79e623 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:33.577148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1014/2039] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:33 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:06:33.580634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c24eb4c-e11b-402e-91a5-8e14698e0241 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:33.594397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c24eb4c-e11b-402e-91a5-8e14698e0241 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:33.605907 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffbc5d1-7522-4b90-859d-1b9b70d3848d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:33.606592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffbc5d1-7522-4b90-859d-1b9b70d3848d admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:33.606940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffbc5d1-7522-4b90-859d-1b9b70d3848d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:33.607327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffbc5d1-7522-4b90-859d-1b9b70d3848d admin admin] PATH_INFO: `/v3/projects/2578bf50dc5948f6a7a9373cb020282b/users/b234abcd2cf5460cb23edc68489d4592/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:33.633525 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bffbc5d1-7522-4b90-859d-1b9b70d3848d admin admin] RBAC: Authorizing `identity:create_grant(project_id=2578bf50dc5948f6a7a9373cb020282b, user_id=b234abcd2cf5460cb23edc68489d4592, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:33.636706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bffbc5d1-7522-4b90-859d-1b9b70d3848d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:33.736118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1026/2040] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:06:33 2022] PUT /identity/v3/projects/2578bf50dc5948f6a7a9373cb020282b/users/b234abcd2cf5460cb23edc68489d4592/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:33.743470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb6ab00-da16-443e-9393-dcee0bbbec4f None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:33.744045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb6ab00-da16-443e-9393-dcee0bbbec4f None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:33.744045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb6ab00-da16-443e-9393-dcee0bbbec4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:33.778154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-ffb6ab00-da16-443e-9393-dcee0bbbec4f None None] MFA Rules not processed for user `b234abcd2cf5460cb23edc68489d4592`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:33.779487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ffb6ab00-da16-443e-9393-dcee0bbbec4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:33.850160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1015/2041] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:33 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:33.862723 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffbc5d1-7522-4b90-859d-1b9b70d3848d admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:33.864456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-bffbc5d1-7522-4b90-859d-1b9b70d3848d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:33.882865 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:33.970160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffbc5d1-7522-4b90-859d-1b9b70d3848d admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:33.974627 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa9c18e-7d26-4642-8286-3381903d87c5 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '69153344fc4b45588bdbcecdfcfd7c3f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:33.975242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa9c18e-7d26-4642-8286-3381903d87c5 service glance] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:33.975523 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa9c18e-7d26-4642-8286-3381903d87c5 service glance] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:33.975801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa9c18e-7d26-4642-8286-3381903d87c5 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:33.985627 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa9c18e-7d26-4642-8286-3381903d87c5 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:33.987079 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa9c18e-7d26-4642-8286-3381903d87c5 service glance] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:33.996217 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1027/2042] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:33 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:35.208996 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd18fdb3-adf6-4f6f-b746-a3b76d88b3f2 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:35.220099 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd18fdb3-adf6-4f6f-b746-a3b76d88b3f2 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:35.220099 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd18fdb3-adf6-4f6f-b746-a3b76d88b3f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:35.245251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-cd18fdb3-adf6-4f6f-b746-a3b76d88b3f2 None None] MFA Rules not processed for user `77de7bb68f3c45d680c8a78dc40df07b`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:35.251061 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-cd18fdb3-adf6-4f6f-b746-a3b76d88b3f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:35.328890 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1016/2043] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:35 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:35.344743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa9c18e-7d26-4642-8286-3381903d87c5 service glance] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:35.349288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-5aa9c18e-7d26-4642-8286-3381903d87c5 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:35.467482 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa9c18e-7d26-4642-8286-3381903d87c5 service glance] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:35.471078 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1700306-3225-4eeb-b263-6a7afdf3a05c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:35.471676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1700306-3225-4eeb-b263-6a7afdf3a05c service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:35.474896 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1700306-3225-4eeb-b263-6a7afdf3a05c service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:35.474896 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1700306-3225-4eeb-b263-6a7afdf3a05c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:35.482230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1700306-3225-4eeb-b263-6a7afdf3a05c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:35.483672 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1700306-3225-4eeb-b263-6a7afdf3a05c service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:35.493446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1028/2044] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:35 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:37.443409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d875a3ee-dc22-4b23-b3da-d7adbe4e1a78 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:37.445110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d875a3ee-dc22-4b23-b3da-d7adbe4e1a78 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:37.445110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d875a3ee-dc22-4b23-b3da-d7adbe4e1a78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:37.465638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-d875a3ee-dc22-4b23-b3da-d7adbe4e1a78 None None] MFA Rules not processed for user `77de7bb68f3c45d680c8a78dc40df07b`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:37.560923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d875a3ee-dc22-4b23-b3da-d7adbe4e1a78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:37.560923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1017/2045] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:37 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:37.692560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3842d3-cdc2-42ca-b000-de8f4b5297d7 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:37.693222 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3842d3-cdc2-42ca-b000-de8f4b5297d7 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:37.693336 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3842d3-cdc2-42ca-b000-de8f4b5297d7 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:37.694643 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1029/2046] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 4 01:06:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:06:37.702937 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2ed242-1837-404e-a0de-830f158c2623 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:37.703365 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2ed242-1837-404e-a0de-830f158c2623 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:37.703667 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2ed242-1837-404e-a0de-830f158c2623 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:37.726068 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-2c2ed242-1837-404e-a0de-830f158c2623 None None] MFA Rules not processed for user `b29f444d56ec49308f7fef16f4d1eea5`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:37.727504 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2c2ed242-1837-404e-a0de-830f158c2623 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:37.762089 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:37.821757 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1018/2047] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 4 01:06:37 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:37.838055 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3d1e02-b09e-4d68-8dd2-f2f59e51a894 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:37.838223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3d1e02-b09e-4d68-8dd2-f2f59e51a894 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:37.838498 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3d1e02-b09e-4d68-8dd2-f2f59e51a894 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:37.840220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1030/2048] 10.222.0.42 () {54 vars in 961 bytes} [Sun Sep 4 01:06:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:06:37.846356 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc605cf-a5f5-48e8-8b43-afd5119e0d82 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:37.847211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc605cf-a5f5-48e8-8b43-afd5119e0d82 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:37.847946 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc605cf-a5f5-48e8-8b43-afd5119e0d82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:37.869735 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-bdc605cf-a5f5-48e8-8b43-afd5119e0d82 None None] MFA Rules not processed for user `b29f444d56ec49308f7fef16f4d1eea5`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:37.870961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-bdc605cf-a5f5-48e8-8b43-afd5119e0d82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:37.881244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1019/2049] 10.222.0.42 () {58 vars in 1058 bytes} [Sun Sep 4 01:06:37 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:37.898030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3d1e02-b09e-4d68-8dd2-f2f59e51a894 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:37.912131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc605cf-a5f5-48e8-8b43-afd5119e0d82 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:37.913754 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-bdc605cf-a5f5-48e8-8b43-afd5119e0d82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:37.916028 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3d1e02-b09e-4d68-8dd2-f2f59e51a894 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:37.923545 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50229d49-03a3-4889-b833-6f36cd46e6d4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b29f444d56ec49308f7fef16f4d1eea5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:37.925425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50229d49-03a3-4889-b833-6f36cd46e6d4 service placement] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:37.925876 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50229d49-03a3-4889-b833-6f36cd46e6d4 service placement] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:37.926309 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50229d49-03a3-4889-b833-6f36cd46e6d4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:37.947079 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50229d49-03a3-4889-b833-6f36cd46e6d4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:37.950916 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50229d49-03a3-4889-b833-6f36cd46e6d4 service placement] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:37.978460 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1031/2050] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:37 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:38.008094 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc605cf-a5f5-48e8-8b43-afd5119e0d82 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:38.017246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0597d436-b294-4779-962d-d585c527e543 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:38.017246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0597d436-b294-4779-962d-d585c527e543 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:38.017246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0597d436-b294-4779-962d-d585c527e543 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:38.017246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0597d436-b294-4779-962d-d585c527e543 admin admin] PATH_INFO: `/v3/users/b6b95b6814cf4cd8b77d87e7810c70e5` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:38.017610 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0597d436-b294-4779-962d-d585c527e543 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6b95b6814cf4cd8b77d87e7810c70e5)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:38.017905 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0597d436-b294-4779-962d-d585c527e543 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:38.382492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0597d436-b294-4779-962d-d585c527e543 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6b95b6814cf4cd8b77d87e7810c70e5'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:38.383273 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0597d436-b294-4779-962d-d585c527e543 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6b95b6814cf4cd8b77d87e7810c70e5'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:38.513560 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0597d436-b294-4779-962d-d585c527e543 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6b95b6814cf4cd8b77d87e7810c70e5'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:38.540502 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0597d436-b294-4779-962d-d585c527e543 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6b95b6814cf4cd8b77d87e7810c70e5'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:38.541309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0597d436-b294-4779-962d-d585c527e543 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6b95b6814cf4cd8b77d87e7810c70e5'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:38.556049 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1020/2051] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:37 2022] DELETE /identity/v3/users/b6b95b6814cf4cd8b77d87e7810c70e5 => generated 0 bytes in 644 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:06:38.825061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50229d49-03a3-4889-b833-6f36cd46e6d4 service placement] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:38.835676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50229d49-03a3-4889-b833-6f36cd46e6d4 service placement] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:38.917629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f2202b-7068-44aa-a79d-501ab63603fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:38.918258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f2202b-7068-44aa-a79d-501ab63603fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:38.918541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f2202b-7068-44aa-a79d-501ab63603fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:38.918813 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f2202b-7068-44aa-a79d-501ab63603fd admin admin] PATH_INFO: `/v3/projects/e07d8b14fef34e04a0410f12e5dbd2fe` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:38.919994 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f2202b-7068-44aa-a79d-501ab63603fd admin admin] RBAC: Authorizing `identity:delete_project(project_id=e07d8b14fef34e04a0410f12e5dbd2fe)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:38.921931 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f2202b-7068-44aa-a79d-501ab63603fd admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:39.023524 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:39.040221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-96f2202b-7068-44aa-a79d-501ab63603fd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e07d8b14fef34e04a0410f12e5dbd2fe', 'request_id': 'req-96f2202b-7068-44aa-a79d-501ab63603fd'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:39.045657 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-96f2202b-7068-44aa-a79d-501ab63603fd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e07d8b14fef34e04a0410f12e5dbd2fe', 'request_id': 'req-96f2202b-7068-44aa-a79d-501ab63603fd'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:39.052048 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-96f2202b-7068-44aa-a79d-501ab63603fd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e07d8b14fef34e04a0410f12e5dbd2fe', 'request_id': 'req-96f2202b-7068-44aa-a79d-501ab63603fd'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:39.132450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-96f2202b-7068-44aa-a79d-501ab63603fd admin admin] The token cache is being invalidate because project e07d8b14fef34e04a0410f12e5dbd2fe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:39.132746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-96f2202b-7068-44aa-a79d-501ab63603fd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:39.133399 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-96f2202b-7068-44aa-a79d-501ab63603fd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:39.134894 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1032/2052] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:38 2022] DELETE /identity/v3/projects/e07d8b14fef34e04a0410f12e5dbd2fe => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:06:39.139864 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a5863b-c92a-4174-9b53-ed3d996a7c30 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:39.140196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a5863b-c92a-4174-9b53-ed3d996a7c30 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:39.140453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a5863b-c92a-4174-9b53-ed3d996a7c30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:39.161068 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-b8a5863b-c92a-4174-9b53-ed3d996a7c30 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:39.162308 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b8a5863b-c92a-4174-9b53-ed3d996a7c30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:39.197406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:39.228096 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1021/2053] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:39.236449 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de0ee22-9726-4e6d-9c74-6968a60df21c None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:39.236746 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de0ee22-9726-4e6d-9c74-6968a60df21c None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:39.238942 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de0ee22-9726-4e6d-9c74-6968a60df21c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:39.258351 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-4de0ee22-9726-4e6d-9c74-6968a60df21c None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:39.262017 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4de0ee22-9726-4e6d-9c74-6968a60df21c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:39.272975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1033/2054] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:39.277287 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a5863b-c92a-4174-9b53-ed3d996a7c30 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:39.287868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a5863b-c92a-4174-9b53-ed3d996a7c30 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:39.293054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb196c3-aafa-45fb-90c6-b2370feb9844 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:39.294300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb196c3-aafa-45fb-90c6-b2370feb9844 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:39.294581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb196c3-aafa-45fb-90c6-b2370feb9844 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:39.294852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb196c3-aafa-45fb-90c6-b2370feb9844 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:39.296051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb196c3-aafa-45fb-90c6-b2370feb9844 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:06:39.296371 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb196c3-aafa-45fb-90c6-b2370feb9844 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:39.298031 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb196c3-aafa-45fb-90c6-b2370feb9844 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:39.324501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1022/2055] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:06:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:06:39.327152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de0ee22-9726-4e6d-9c74-6968a60df21c None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:39.339198 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de0ee22-9726-4e6d-9c74-6968a60df21c None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:39.344304 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892c642c-386c-42b7-973c-9a00caa90497 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:39.344913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892c642c-386c-42b7-973c-9a00caa90497 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:39.345239 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892c642c-386c-42b7-973c-9a00caa90497 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:39.356562 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892c642c-386c-42b7-973c-9a00caa90497 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:39.356562 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892c642c-386c-42b7-973c-9a00caa90497 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:39.356562 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892c642c-386c-42b7-973c-9a00caa90497 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:39.449360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1034/2056] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:06:39 2022] POST /identity/v3/projects => generated 385 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:06:39.453722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb196c3-aafa-45fb-90c6-b2370feb9844 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:39.468756 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb196c3-aafa-45fb-90c6-b2370feb9844 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:39.536417 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b1ac12-b1f6-465b-a86a-b696b2622cca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:39.537048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b1ac12-b1f6-465b-a86a-b696b2622cca admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:39.537372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b1ac12-b1f6-465b-a86a-b696b2622cca admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:39.537648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b1ac12-b1f6-465b-a86a-b696b2622cca admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:39.538313 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b1ac12-b1f6-465b-a86a-b696b2622cca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:39.540225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b1ac12-b1f6-465b-a86a-b696b2622cca admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:39.648409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1023/2057] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:06:39 2022] POST /identity/v3/users => generated 293 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:06:39.652918 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892c642c-386c-42b7-973c-9a00caa90497 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:39.664235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892c642c-386c-42b7-973c-9a00caa90497 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:39.668987 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939ae8d5-e591-4877-97f2-334c3ec69d64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:39.669621 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939ae8d5-e591-4877-97f2-334c3ec69d64 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:39.669898 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939ae8d5-e591-4877-97f2-334c3ec69d64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:39.670171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939ae8d5-e591-4877-97f2-334c3ec69d64 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:39.670815 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939ae8d5-e591-4877-97f2-334c3ec69d64 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:39.672430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939ae8d5-e591-4877-97f2-334c3ec69d64 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:39.683500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1035/2058] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:06:39.687523 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b1ac12-b1f6-465b-a86a-b696b2622cca admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:39.698279 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b1ac12-b1f6-465b-a86a-b696b2622cca admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:39.703212 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a8c3d7-0583-48be-b1e9-8c151a0146e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:39.703867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8c3d7-0583-48be-b1e9-8c151a0146e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:39.704149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8c3d7-0583-48be-b1e9-8c151a0146e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:39.704422 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a8c3d7-0583-48be-b1e9-8c151a0146e0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:39.705087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a8c3d7-0583-48be-b1e9-8c151a0146e0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:39.707278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a8c3d7-0583-48be-b1e9-8c151a0146e0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:39.729655 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1024/2059] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:39 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:06:39.734011 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939ae8d5-e591-4877-97f2-334c3ec69d64 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:39.746429 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939ae8d5-e591-4877-97f2-334c3ec69d64 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:39.750861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81eac597-a333-4fe8-a69c-fb081faa0ec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:39.751513 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81eac597-a333-4fe8-a69c-fb081faa0ec0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:39.751837 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81eac597-a333-4fe8-a69c-fb081faa0ec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:39.752111 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81eac597-a333-4fe8-a69c-fb081faa0ec0 admin admin] PATH_INFO: `/v3/projects/07a7237c4f0c4f3e8e43aa14ec0a9064/users/f55f74f18e4c4e77b2d9d8b68ae0ea90/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:39.767564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81eac597-a333-4fe8-a69c-fb081faa0ec0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=07a7237c4f0c4f3e8e43aa14ec0a9064, user_id=f55f74f18e4c4e77b2d9d8b68ae0ea90, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:39.771937 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81eac597-a333-4fe8-a69c-fb081faa0ec0 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:39.856475 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1036/2060] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:06:39 2022] PUT /identity/v3/projects/07a7237c4f0c4f3e8e43aa14ec0a9064/users/f55f74f18e4c4e77b2d9d8b68ae0ea90/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:39.867512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96788cf-b7b8-457b-8fdf-ec7206dc0d2d None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:39.867805 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96788cf-b7b8-457b-8fdf-ec7206dc0d2d None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:39.868083 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96788cf-b7b8-457b-8fdf-ec7206dc0d2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:39.897724 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-d96788cf-b7b8-457b-8fdf-ec7206dc0d2d None None] MFA Rules not processed for user `f55f74f18e4c4e77b2d9d8b68ae0ea90`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:39.898986 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d96788cf-b7b8-457b-8fdf-ec7206dc0d2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:39.974455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1025/2061] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:39 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:39.985689 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81eac597-a333-4fe8-a69c-fb081faa0ec0 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:39.987853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-81eac597-a333-4fe8-a69c-fb081faa0ec0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:40.070805 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81eac597-a333-4fe8-a69c-fb081faa0ec0 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:40.077739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b52e8f-d70f-428c-976c-90ce1d21ec4a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:40.078344 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b52e8f-d70f-428c-976c-90ce1d21ec4a service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:40.078622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b52e8f-d70f-428c-976c-90ce1d21ec4a service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:40.078895 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b52e8f-d70f-428c-976c-90ce1d21ec4a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:40.088934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b52e8f-d70f-428c-976c-90ce1d21ec4a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:40.090907 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b52e8f-d70f-428c-976c-90ce1d21ec4a service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:40.100789 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1037/2062] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:39 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:40.839818 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65edb98b-254c-4d4e-86ae-f4fc401cb088 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:40.840318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65edb98b-254c-4d4e-86ae-f4fc401cb088 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:40.851406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65edb98b-254c-4d4e-86ae-f4fc401cb088 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:40.859930 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-65edb98b-254c-4d4e-86ae-f4fc401cb088 None None] MFA Rules not processed for user `70cef7c271e046a3b49230057cc2d28f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:40.861423 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-65edb98b-254c-4d4e-86ae-f4fc401cb088 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:40.894325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:40.920092 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1026/2063] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 4 01:06:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:40.962652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b52e8f-d70f-428c-976c-90ce1d21ec4a service nova] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:40.964709 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-46b52e8f-d70f-428c-976c-90ce1d21ec4a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:41.012472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:DELETE=6,keystone:SELECT=91,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:41.041405 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b52e8f-d70f-428c-976c-90ce1d21ec4a service nova] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:41.046393 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f1a1d9-5ebc-4084-9246-026e2360063f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '123ab6ff796c4e599ce1d6f4095bd898', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:41.046990 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f1a1d9-5ebc-4084-9246-026e2360063f service swift] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:41.047269 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f1a1d9-5ebc-4084-9246-026e2360063f service swift] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:41.047540 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f1a1d9-5ebc-4084-9246-026e2360063f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:41.056055 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f1a1d9-5ebc-4084-9246-026e2360063f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:41.057671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f1a1d9-5ebc-4084-9246-026e2360063f service swift] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:41.066523 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1038/2064] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 4 01:06:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 04 01:06:42.071611 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52624654-71e1-4c2d-89b0-18591097900a None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:42.072107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52624654-71e1-4c2d-89b0-18591097900a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:42.072206 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52624654-71e1-4c2d-89b0-18591097900a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:42.092283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-52624654-71e1-4c2d-89b0-18591097900a None None] MFA Rules not processed for user `70cef7c271e046a3b49230057cc2d28f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:42.093559 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-52624654-71e1-4c2d-89b0-18591097900a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:42.101335 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1027/2065] 10.222.0.42 () {58 vars in 967 bytes} [Sun Sep 4 01:06:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:42.114483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f1a1d9-5ebc-4084-9246-026e2360063f service swift] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:42.128853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f1a1d9-5ebc-4084-9246-026e2360063f service swift] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:42.135443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1409d46b-70d5-4d3b-91a8-be2bc071c695 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '123ab6ff796c4e599ce1d6f4095bd898', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:42.136206 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1409d46b-70d5-4d3b-91a8-be2bc071c695 service swift] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:42.136586 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1409d46b-70d5-4d3b-91a8-be2bc071c695 service swift] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:42.137025 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1409d46b-70d5-4d3b-91a8-be2bc071c695 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:42.159680 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1409d46b-70d5-4d3b-91a8-be2bc071c695 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:42.159680 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1409d46b-70d5-4d3b-91a8-be2bc071c695 service swift] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:42.166371 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1039/2066] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 4 01:06:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 04 01:06:42.829358 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ad173a-4e62-491b-8088-dd50aab4bf92 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:42.829644 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ad173a-4e62-491b-8088-dd50aab4bf92 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:42.829917 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ad173a-4e62-491b-8088-dd50aab4bf92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:42.849660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-47ad173a-4e62-491b-8088-dd50aab4bf92 None None] MFA Rules not processed for user `70cef7c271e046a3b49230057cc2d28f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:42.850864 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-47ad173a-4e62-491b-8088-dd50aab4bf92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:42.858508 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1028/2067] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 4 01:06:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:42.872143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1409d46b-70d5-4d3b-91a8-be2bc071c695 service swift] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:42.883425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1409d46b-70d5-4d3b-91a8-be2bc071c695 service swift] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:42.895890 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3707b3-552f-492c-abcf-f8ff19711a68 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '123ab6ff796c4e599ce1d6f4095bd898', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:42.896530 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3707b3-552f-492c-abcf-f8ff19711a68 service swift] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:42.897037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3707b3-552f-492c-abcf-f8ff19711a68 service swift] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:42.897548 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d3707b3-552f-492c-abcf-f8ff19711a68 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:42.907076 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3707b3-552f-492c-abcf-f8ff19711a68 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:42.908664 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d3707b3-552f-492c-abcf-f8ff19711a68 service swift] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:42.918783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1040/2068] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 4 01:06:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 04 01:06:43.542207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24bb2a9-bc4b-49b6-b216-2d2b42e2d1fa None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:43.542736 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24bb2a9-bc4b-49b6-b216-2d2b42e2d1fa None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:43.542926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24bb2a9-bc4b-49b6-b216-2d2b42e2d1fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:43.563051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-b24bb2a9-bc4b-49b6-b216-2d2b42e2d1fa None None] MFA Rules not processed for user `70cef7c271e046a3b49230057cc2d28f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:43.564250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b24bb2a9-bc4b-49b6-b216-2d2b42e2d1fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:43.571834 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1029/2069] 10.222.0.42 () {58 vars in 967 bytes} [Sun Sep 4 01:06:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:43.584334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3707b3-552f-492c-abcf-f8ff19711a68 service swift] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:43.594852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d3707b3-552f-492c-abcf-f8ff19711a68 service swift] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:43.600483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af008857-dbde-439e-bef1-837dea52bf88 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '123ab6ff796c4e599ce1d6f4095bd898', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:43.601209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af008857-dbde-439e-bef1-837dea52bf88 service swift] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:43.601588 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af008857-dbde-439e-bef1-837dea52bf88 service swift] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:43.601943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af008857-dbde-439e-bef1-837dea52bf88 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:43.610121 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af008857-dbde-439e-bef1-837dea52bf88 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:43.612628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af008857-dbde-439e-bef1-837dea52bf88 service swift] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:43.621434 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1041/2070] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 4 01:06:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 04 01:06:47.327234 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24bb2a9-bc4b-49b6-b216-2d2b42e2d1fa None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:47.329180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b24bb2a9-bc4b-49b6-b216-2d2b42e2d1fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:47.417754 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24bb2a9-bc4b-49b6-b216-2d2b42e2d1fa None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:47.422555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52898e2e-dbe6-46d5-b49b-1688f4387020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:47.423380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52898e2e-dbe6-46d5-b49b-1688f4387020 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:47.423663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52898e2e-dbe6-46d5-b49b-1688f4387020 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:47.424123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52898e2e-dbe6-46d5-b49b-1688f4387020 admin admin] PATH_INFO: `/v3/users/af874cda15384e2f9160ccf6866296f6` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:47.426229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52898e2e-dbe6-46d5-b49b-1688f4387020 admin admin] RBAC: Authorizing `identity:delete_user(user_id=af874cda15384e2f9160ccf6866296f6)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:47.428192 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52898e2e-dbe6-46d5-b49b-1688f4387020 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:47.590868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-52898e2e-dbe6-46d5-b49b-1688f4387020 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af874cda15384e2f9160ccf6866296f6'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:47.591668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-52898e2e-dbe6-46d5-b49b-1688f4387020 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af874cda15384e2f9160ccf6866296f6'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:47.674422 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-52898e2e-dbe6-46d5-b49b-1688f4387020 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af874cda15384e2f9160ccf6866296f6'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:47.692597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-52898e2e-dbe6-46d5-b49b-1688f4387020 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af874cda15384e2f9160ccf6866296f6'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:47.693431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-52898e2e-dbe6-46d5-b49b-1688f4387020 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af874cda15384e2f9160ccf6866296f6'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:47.720875 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1030/2071] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:47 2022] DELETE /identity/v3/users/af874cda15384e2f9160ccf6866296f6 => generated 0 bytes in 394 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:47.726834 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e8b85c-8d26-4713-b56c-6f22f57fde3d None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:47.727932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e8b85c-8d26-4713-b56c-6f22f57fde3d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:47.728208 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e8b85c-8d26-4713-b56c-6f22f57fde3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:47.750156 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-98e8b85c-8d26-4713-b56c-6f22f57fde3d None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:47.751372 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-98e8b85c-8d26-4713-b56c-6f22f57fde3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:47.812975 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1042/2072] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:47.824358 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52898e2e-dbe6-46d5-b49b-1688f4387020 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:47.825838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-52898e2e-dbe6-46d5-b49b-1688f4387020 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:47.867673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:47.883059 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52898e2e-dbe6-46d5-b49b-1688f4387020 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:47.886802 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e3e9fa-c524-498c-a724-491df8a6a3ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:47.887750 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e3e9fa-c524-498c-a724-491df8a6a3ab service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:47.888046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e3e9fa-c524-498c-a724-491df8a6a3ab service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:47.888315 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e3e9fa-c524-498c-a724-491df8a6a3ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:47.898614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e3e9fa-c524-498c-a724-491df8a6a3ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:47.900036 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e3e9fa-c524-498c-a724-491df8a6a3ab service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:47.909596 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1031/2073] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:48.431484 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e8b85c-8d26-4713-b56c-6f22f57fde3d None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:48.432171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-98e8b85c-8d26-4713-b56c-6f22f57fde3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:48.451755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e8b85c-8d26-4713-b56c-6f22f57fde3d None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:48.458265 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccebffca-ef1e-440e-ba1e-1cfadf0539a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:48.460291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccebffca-ef1e-440e-ba1e-1cfadf0539a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:48.461007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccebffca-ef1e-440e-ba1e-1cfadf0539a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:48.461927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccebffca-ef1e-440e-ba1e-1cfadf0539a7 admin admin] PATH_INFO: `/v3/projects/528d5c15cc47445dbe46761067a1c54a` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:48.464538 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccebffca-ef1e-440e-ba1e-1cfadf0539a7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=528d5c15cc47445dbe46761067a1c54a)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:48.467188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccebffca-ef1e-440e-ba1e-1cfadf0539a7 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:48.587420 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-ccebffca-ef1e-440e-ba1e-1cfadf0539a7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '528d5c15cc47445dbe46761067a1c54a', 'request_id': 'req-ccebffca-ef1e-440e-ba1e-1cfadf0539a7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:48.594976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-ccebffca-ef1e-440e-ba1e-1cfadf0539a7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '528d5c15cc47445dbe46761067a1c54a', 'request_id': 'req-ccebffca-ef1e-440e-ba1e-1cfadf0539a7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:48.602112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-ccebffca-ef1e-440e-ba1e-1cfadf0539a7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '528d5c15cc47445dbe46761067a1c54a', 'request_id': 'req-ccebffca-ef1e-440e-ba1e-1cfadf0539a7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:48.698415 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-ccebffca-ef1e-440e-ba1e-1cfadf0539a7 admin admin] The token cache is being invalidate because project 528d5c15cc47445dbe46761067a1c54a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:48.698842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-ccebffca-ef1e-440e-ba1e-1cfadf0539a7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:48.699809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-ccebffca-ef1e-440e-ba1e-1cfadf0539a7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:48.701518 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1043/2074] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:48 2022] DELETE /identity/v3/projects/528d5c15cc47445dbe46761067a1c54a => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:49.452183 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e3e9fa-c524-498c-a724-491df8a6a3ab service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:49.452822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-18e3e9fa-c524-498c-a724-491df8a6a3ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:49.466696 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e3e9fa-c524-498c-a724-491df8a6a3ab service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:49.528843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ec4973-776b-4978-85c0-141ae9050528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:49.529526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ec4973-776b-4978-85c0-141ae9050528 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:49.529806 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ec4973-776b-4978-85c0-141ae9050528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:49.530163 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ec4973-776b-4978-85c0-141ae9050528 admin admin] PATH_INFO: `/v3/users/f55f74f18e4c4e77b2d9d8b68ae0ea90` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:49.531343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ec4973-776b-4978-85c0-141ae9050528 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f55f74f18e4c4e77b2d9d8b68ae0ea90)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:49.533294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ec4973-776b-4978-85c0-141ae9050528 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:49.774710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-f6ec4973-776b-4978-85c0-141ae9050528 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f55f74f18e4c4e77b2d9d8b68ae0ea90'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:49.775373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-f6ec4973-776b-4978-85c0-141ae9050528 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f55f74f18e4c4e77b2d9d8b68ae0ea90'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:49.822955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-f6ec4973-776b-4978-85c0-141ae9050528 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f55f74f18e4c4e77b2d9d8b68ae0ea90'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:49.837358 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-f6ec4973-776b-4978-85c0-141ae9050528 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f55f74f18e4c4e77b2d9d8b68ae0ea90'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:49.838228 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-f6ec4973-776b-4978-85c0-141ae9050528 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f55f74f18e4c4e77b2d9d8b68ae0ea90'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:49.852578 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1032/2075] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:06:49 2022] DELETE /identity/v3/users/f55f74f18e4c4e77b2d9d8b68ae0ea90 => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:49.857180 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2804ab4b-cd28-41a3-8640-d1be375b590f None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:49.857639 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2804ab4b-cd28-41a3-8640-d1be375b590f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:49.857803 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2804ab4b-cd28-41a3-8640-d1be375b590f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:49.878688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-2804ab4b-cd28-41a3-8640-d1be375b590f None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:49.879893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-2804ab4b-cd28-41a3-8640-d1be375b590f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:49.890484 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:49.964403 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1044/2076] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:49.974109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ec4973-776b-4978-85c0-141ae9050528 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:49.975626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f6ec4973-776b-4978-85c0-141ae9050528 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:50.052584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ec4973-776b-4978-85c0-141ae9050528 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:50.058503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f781d29e-3d55-4177-9aa1-270d2ef255a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:50.059124 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f781d29e-3d55-4177-9aa1-270d2ef255a8 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:50.059406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f781d29e-3d55-4177-9aa1-270d2ef255a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:50.059680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f781d29e-3d55-4177-9aa1-270d2ef255a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:50.078598 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f781d29e-3d55-4177-9aa1-270d2ef255a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:50.080170 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f781d29e-3d55-4177-9aa1-270d2ef255a8 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:50.091672 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1033/2077] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:06:50.597311 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2804ab4b-cd28-41a3-8640-d1be375b590f None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:50.597454 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-2804ab4b-cd28-41a3-8640-d1be375b590f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:50.609343 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2804ab4b-cd28-41a3-8640-d1be375b590f None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:50.614993 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb63c09-8ee0-4777-8c12-76a7390791e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:50.615628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb63c09-8ee0-4777-8c12-76a7390791e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:50.615906 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb63c09-8ee0-4777-8c12-76a7390791e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:50.616177 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb63c09-8ee0-4777-8c12-76a7390791e0 admin admin] PATH_INFO: `/v3/projects/07a7237c4f0c4f3e8e43aa14ec0a9064` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:50.617337 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb63c09-8ee0-4777-8c12-76a7390791e0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=07a7237c4f0c4f3e8e43aa14ec0a9064)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:50.619253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb63c09-8ee0-4777-8c12-76a7390791e0 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:50.780409 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bdb63c09-8ee0-4777-8c12-76a7390791e0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '07a7237c4f0c4f3e8e43aa14ec0a9064', 'request_id': 'req-bdb63c09-8ee0-4777-8c12-76a7390791e0'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:50.787237 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bdb63c09-8ee0-4777-8c12-76a7390791e0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '07a7237c4f0c4f3e8e43aa14ec0a9064', 'request_id': 'req-bdb63c09-8ee0-4777-8c12-76a7390791e0'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:50.793593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bdb63c09-8ee0-4777-8c12-76a7390791e0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '07a7237c4f0c4f3e8e43aa14ec0a9064', 'request_id': 'req-bdb63c09-8ee0-4777-8c12-76a7390791e0'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:50.862649 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bdb63c09-8ee0-4777-8c12-76a7390791e0 admin admin] The token cache is being invalidate because project 07a7237c4f0c4f3e8e43aa14ec0a9064 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:06:50.863251 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bdb63c09-8ee0-4777-8c12-76a7390791e0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:50.863501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bdb63c09-8ee0-4777-8c12-76a7390791e0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:06:50.864995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1045/2078] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:06:50 2022] DELETE /identity/v3/projects/07a7237c4f0c4f3e8e43aa14ec0a9064 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:50.870091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fb3e52-4710-4a83-a22a-f9a3dbd14a2a None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:50.870514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fb3e52-4710-4a83-a22a-f9a3dbd14a2a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:50.870641 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fb3e52-4710-4a83-a22a-f9a3dbd14a2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:50.879053 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:50.892378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-05fb3e52-4710-4a83-a22a-f9a3dbd14a2a None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:50.894240 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-05fb3e52-4710-4a83-a22a-f9a3dbd14a2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:50.953769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1034/2079] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:50.963913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efa0602-f3fd-4816-b7ce-0f14ed670150 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:50.964296 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efa0602-f3fd-4816-b7ce-0f14ed670150 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:50.964592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efa0602-f3fd-4816-b7ce-0f14ed670150 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:50.985922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-1efa0602-f3fd-4816-b7ce-0f14ed670150 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:50.987169 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-1efa0602-f3fd-4816-b7ce-0f14ed670150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:50.995350 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1046/2080] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:50.999647 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fb3e52-4710-4a83-a22a-f9a3dbd14a2a None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:51.010214 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fb3e52-4710-4a83-a22a-f9a3dbd14a2a None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:51.014887 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56717fc3-d625-4936-bcd5-bd26ababdb90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:51.015477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56717fc3-d625-4936-bcd5-bd26ababdb90 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:51.015759 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56717fc3-d625-4936-bcd5-bd26ababdb90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:51.016032 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56717fc3-d625-4936-bcd5-bd26ababdb90 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:51.018451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56717fc3-d625-4936-bcd5-bd26ababdb90 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:06:51.018451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56717fc3-d625-4936-bcd5-bd26ababdb90 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:51.019491 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56717fc3-d625-4936-bcd5-bd26ababdb90 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:51.030804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1035/2081] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:06:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:06:51.037152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efa0602-f3fd-4816-b7ce-0f14ed670150 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:51.045117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efa0602-f3fd-4816-b7ce-0f14ed670150 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:51.050072 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a82582-a14e-49d8-9029-fb816a4626dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:51.050663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a82582-a14e-49d8-9029-fb816a4626dd admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:51.050943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a82582-a14e-49d8-9029-fb816a4626dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:51.051280 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a82582-a14e-49d8-9029-fb816a4626dd admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:51.051992 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a82582-a14e-49d8-9029-fb816a4626dd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:51.053911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a82582-a14e-49d8-9029-fb816a4626dd admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:51.169455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1047/2082] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:06:51 2022] POST /identity/v3/projects => generated 367 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:06:51.170824 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56717fc3-d625-4936-bcd5-bd26ababdb90 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:51.180878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56717fc3-d625-4936-bcd5-bd26ababdb90 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:51.241038 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c179395-8ed5-41e9-84a8-e8dbb34ce9a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:51.241683 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c179395-8ed5-41e9-84a8-e8dbb34ce9a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:51.242135 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c179395-8ed5-41e9-84a8-e8dbb34ce9a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:51.242467 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c179395-8ed5-41e9-84a8-e8dbb34ce9a8 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:51.243107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c179395-8ed5-41e9-84a8-e8dbb34ce9a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:51.245010 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c179395-8ed5-41e9-84a8-e8dbb34ce9a8 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:51.330754 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1036/2083] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:06:51 2022] POST /identity/v3/users => generated 284 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:06:51.334782 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a82582-a14e-49d8-9029-fb816a4626dd admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:51.344967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a82582-a14e-49d8-9029-fb816a4626dd admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:51.351435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca90974-9fd1-4f14-8891-1461055d2eb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:51.351435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca90974-9fd1-4f14-8891-1461055d2eb4 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:51.351435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca90974-9fd1-4f14-8891-1461055d2eb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:51.351435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca90974-9fd1-4f14-8891-1461055d2eb4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:51.351914 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca90974-9fd1-4f14-8891-1461055d2eb4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:51.353546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca90974-9fd1-4f14-8891-1461055d2eb4 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:51.364491 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1048/2084] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:06:51.368507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c179395-8ed5-41e9-84a8-e8dbb34ce9a8 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:51.378563 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c179395-8ed5-41e9-84a8-e8dbb34ce9a8 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:51.383942 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb712591-3276-4be7-8874-41b7196b74b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:51.384746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb712591-3276-4be7-8874-41b7196b74b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:51.385060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb712591-3276-4be7-8874-41b7196b74b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:51.385366 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb712591-3276-4be7-8874-41b7196b74b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:51.386021 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb712591-3276-4be7-8874-41b7196b74b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:51.387622 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb712591-3276-4be7-8874-41b7196b74b0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:51.399530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1037/2085] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:06:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:06:51.405098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca90974-9fd1-4f14-8891-1461055d2eb4 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:51.415791 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca90974-9fd1-4f14-8891-1461055d2eb4 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:51.420285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03f45b4-ca36-49a2-b434-3700ec960520 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:51.420933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03f45b4-ca36-49a2-b434-3700ec960520 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:51.421359 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03f45b4-ca36-49a2-b434-3700ec960520 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:51.421657 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03f45b4-ca36-49a2-b434-3700ec960520 admin admin] PATH_INFO: `/v3/projects/abb283f343024bcd8dde457024425a7b/users/a04672067fa1419c8450fb2d0ac48379/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:51.437357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03f45b4-ca36-49a2-b434-3700ec960520 admin admin] RBAC: Authorizing `identity:create_grant(project_id=abb283f343024bcd8dde457024425a7b, user_id=a04672067fa1419c8450fb2d0ac48379, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:51.440193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03f45b4-ca36-49a2-b434-3700ec960520 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:51.541858 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1049/2086] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:06:51 2022] PUT /identity/v3/projects/abb283f343024bcd8dde457024425a7b/users/a04672067fa1419c8450fb2d0ac48379/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:06:51.552414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f85f9c-2c8a-4dfd-934b-8fd4cbed4bdc None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:51.552701 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f85f9c-2c8a-4dfd-934b-8fd4cbed4bdc None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:51.552976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f85f9c-2c8a-4dfd-934b-8fd4cbed4bdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:51.581317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-21f85f9c-2c8a-4dfd-934b-8fd4cbed4bdc None None] MFA Rules not processed for user `a04672067fa1419c8450fb2d0ac48379`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:06:51.582669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-21f85f9c-2c8a-4dfd-934b-8fd4cbed4bdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:51.641066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:51.646978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1038/2087] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:06:51 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:06:51.664236 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03f45b4-ca36-49a2-b434-3700ec960520 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:06:51.666977 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c03f45b4-ca36-49a2-b434-3700ec960520 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:06:51.700508 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:06:51.729791 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03f45b4-ca36-49a2-b434-3700ec960520 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:06:51.733544 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0292ad0c-93cc-4139-9137-03d807861f84 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:06:51.734151 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0292ad0c-93cc-4139-9137-03d807861f84 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:06:51.734432 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0292ad0c-93cc-4139-9137-03d807861f84 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:06:51.734708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0292ad0c-93cc-4139-9137-03d807861f84 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:06:51.744039 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0292ad0c-93cc-4139-9137-03d807861f84 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:06:51.745341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0292ad0c-93cc-4139-9137-03d807861f84 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:06:51.755122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1050/2088] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:06:51 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:07:01.731181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:02.125367 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=13 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:04.853418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f85f9c-2c8a-4dfd-934b-8fd4cbed4bdc None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:04.927453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f85f9c-2c8a-4dfd-934b-8fd4cbed4bdc None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:04.932209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3c3048-0325-436c-8299-4705951f7db8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:04.932954 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3c3048-0325-436c-8299-4705951f7db8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:04.933196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3c3048-0325-436c-8299-4705951f7db8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:04.933421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3c3048-0325-436c-8299-4705951f7db8 admin admin] PATH_INFO: `/v3/users/a04672067fa1419c8450fb2d0ac48379` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:04.934579 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3c3048-0325-436c-8299-4705951f7db8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a04672067fa1419c8450fb2d0ac48379)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:04.936501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3c3048-0325-436c-8299-4705951f7db8 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:05.149393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-2b3c3048-0325-436c-8299-4705951f7db8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a04672067fa1419c8450fb2d0ac48379'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:05.150066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-2b3c3048-0325-436c-8299-4705951f7db8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a04672067fa1419c8450fb2d0ac48379'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:05.228409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-2b3c3048-0325-436c-8299-4705951f7db8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a04672067fa1419c8450fb2d0ac48379'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:05.243195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-2b3c3048-0325-436c-8299-4705951f7db8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a04672067fa1419c8450fb2d0ac48379'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:05.243903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-2b3c3048-0325-436c-8299-4705951f7db8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a04672067fa1419c8450fb2d0ac48379'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:05.261195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1039/2089] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:07:04 2022] DELETE /identity/v3/users/a04672067fa1419c8450fb2d0ac48379 => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:07:05.272776 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b3414c-9a23-494c-a69c-66e75071a609 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:05.273632 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b3414c-9a23-494c-a69c-66e75071a609 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:05.274141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b3414c-9a23-494c-a69c-66e75071a609 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:05.306162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-08b3414c-9a23-494c-a69c-66e75071a609 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:05.307577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-08b3414c-9a23-494c-a69c-66e75071a609 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:05.386689 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1051/2090] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:05.393138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3c3048-0325-436c-8299-4705951f7db8 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:05.395158 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2b3c3048-0325-436c-8299-4705951f7db8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:05.456566 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3c3048-0325-436c-8299-4705951f7db8 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:05.467412 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6cbb6c-cfdc-4477-a585-bce794059ccd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:05.468120 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6cbb6c-cfdc-4477-a585-bce794059ccd service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:05.468573 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6cbb6c-cfdc-4477-a585-bce794059ccd service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:05.468961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6cbb6c-cfdc-4477-a585-bce794059ccd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:05.480867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6cbb6c-cfdc-4477-a585-bce794059ccd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:05.482620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6cbb6c-cfdc-4477-a585-bce794059ccd service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:05.492836 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1040/2091] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:07:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:07:05.989208 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b3414c-9a23-494c-a69c-66e75071a609 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:05.990813 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-08b3414c-9a23-494c-a69c-66e75071a609 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:06.002852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b3414c-9a23-494c-a69c-66e75071a609 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:06.008595 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b35ecb9-f8b8-4792-b10e-e6d6e6f737d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:06.009338 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b35ecb9-f8b8-4792-b10e-e6d6e6f737d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:06.009690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b35ecb9-f8b8-4792-b10e-e6d6e6f737d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:06.010059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b35ecb9-f8b8-4792-b10e-e6d6e6f737d4 admin admin] PATH_INFO: `/v3/projects/abb283f343024bcd8dde457024425a7b` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:06.012335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b35ecb9-f8b8-4792-b10e-e6d6e6f737d4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=abb283f343024bcd8dde457024425a7b)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:06.014494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b35ecb9-f8b8-4792-b10e-e6d6e6f737d4 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:06.136576 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4b35ecb9-f8b8-4792-b10e-e6d6e6f737d4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'abb283f343024bcd8dde457024425a7b', 'request_id': 'req-4b35ecb9-f8b8-4792-b10e-e6d6e6f737d4'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:06.142417 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4b35ecb9-f8b8-4792-b10e-e6d6e6f737d4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'abb283f343024bcd8dde457024425a7b', 'request_id': 'req-4b35ecb9-f8b8-4792-b10e-e6d6e6f737d4'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:06.150125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4b35ecb9-f8b8-4792-b10e-e6d6e6f737d4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'abb283f343024bcd8dde457024425a7b', 'request_id': 'req-4b35ecb9-f8b8-4792-b10e-e6d6e6f737d4'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:06.240196 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4b35ecb9-f8b8-4792-b10e-e6d6e6f737d4 admin admin] The token cache is being invalidate because project abb283f343024bcd8dde457024425a7b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:07:06.240510 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4b35ecb9-f8b8-4792-b10e-e6d6e6f737d4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:06.241237 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4b35ecb9-f8b8-4792-b10e-e6d6e6f737d4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:06.242788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1052/2092] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:07:05 2022] DELETE /identity/v3/projects/abb283f343024bcd8dde457024425a7b => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:07:06.247695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7766a649-b972-4d04-acb3-6814a7a2ab86 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:06.248139 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7766a649-b972-4d04-acb3-6814a7a2ab86 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:06.248225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7766a649-b972-4d04-acb3-6814a7a2ab86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:06.268022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-7766a649-b972-4d04-acb3-6814a7a2ab86 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:06.269983 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-7766a649-b972-4d04-acb3-6814a7a2ab86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:06.286851 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:06.330746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1041/2093] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:06.339484 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c207e125-9868-4319-9622-bbe9fedf34b9 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:06.339968 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c207e125-9868-4319-9622-bbe9fedf34b9 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:06.340060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c207e125-9868-4319-9622-bbe9fedf34b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:06.360576 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-c207e125-9868-4319-9622-bbe9fedf34b9 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:06.361836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c207e125-9868-4319-9622-bbe9fedf34b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:06.377179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1053/2094] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:06.383679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7766a649-b972-4d04-acb3-6814a7a2ab86 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:06.393375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7766a649-b972-4d04-acb3-6814a7a2ab86 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:06.398230 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b3fbed-0efe-4b47-b8be-27fc0f06982c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:06.398829 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b3fbed-0efe-4b47-b8be-27fc0f06982c admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:06.399109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b3fbed-0efe-4b47-b8be-27fc0f06982c admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:06.399378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b3fbed-0efe-4b47-b8be-27fc0f06982c admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:06.400026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b3fbed-0efe-4b47-b8be-27fc0f06982c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:07:06.400329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b3fbed-0efe-4b47-b8be-27fc0f06982c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:06.401940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b3fbed-0efe-4b47-b8be-27fc0f06982c admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:06.414334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1042/2095] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:07:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:07:06.418737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c207e125-9868-4319-9622-bbe9fedf34b9 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:06.435168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c207e125-9868-4319-9622-bbe9fedf34b9 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:06.440123 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d23cf9f-1a2c-436d-b163-32a980da63d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:06.440732 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d23cf9f-1a2c-436d-b163-32a980da63d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:06.441811 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d23cf9f-1a2c-436d-b163-32a980da63d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:06.442090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d23cf9f-1a2c-436d-b163-32a980da63d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:06.442747 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d23cf9f-1a2c-436d-b163-32a980da63d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:06.444644 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d23cf9f-1a2c-436d-b163-32a980da63d2 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:06.559990 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1054/2096] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:07:06 2022] POST /identity/v3/projects => generated 379 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:07:06.564066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b3fbed-0efe-4b47-b8be-27fc0f06982c admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:06.575545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b3fbed-0efe-4b47-b8be-27fc0f06982c admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:06.634737 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8399759b-fc94-481c-9df6-fcf9cf73a920 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:06.635342 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8399759b-fc94-481c-9df6-fcf9cf73a920 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:06.635619 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8399759b-fc94-481c-9df6-fcf9cf73a920 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:06.635916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8399759b-fc94-481c-9df6-fcf9cf73a920 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:06.636555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8399759b-fc94-481c-9df6-fcf9cf73a920 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:06.638483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8399759b-fc94-481c-9df6-fcf9cf73a920 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:06.720283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1043/2097] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:07:06 2022] POST /identity/v3/users => generated 290 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:07:06.724884 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d23cf9f-1a2c-436d-b163-32a980da63d2 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:06.735522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d23cf9f-1a2c-436d-b163-32a980da63d2 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:06.740602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31aaea9d-49b8-4723-b8a4-2182b8863226 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:06.741261 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aaea9d-49b8-4723-b8a4-2182b8863226 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:06.741541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aaea9d-49b8-4723-b8a4-2182b8863226 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:06.741810 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aaea9d-49b8-4723-b8a4-2182b8863226 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:06.742445 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31aaea9d-49b8-4723-b8a4-2182b8863226 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:06.744038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31aaea9d-49b8-4723-b8a4-2182b8863226 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:06.753741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1055/2098] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:07:06 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:07:06.757795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8399759b-fc94-481c-9df6-fcf9cf73a920 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:06.768130 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8399759b-fc94-481c-9df6-fcf9cf73a920 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:06.773902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177bc56f-b9cd-483b-8c2b-4636a0a78714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:06.774528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177bc56f-b9cd-483b-8c2b-4636a0a78714 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:06.774812 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177bc56f-b9cd-483b-8c2b-4636a0a78714 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:06.775087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177bc56f-b9cd-483b-8c2b-4636a0a78714 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:06.775751 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177bc56f-b9cd-483b-8c2b-4636a0a78714 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:06.777410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177bc56f-b9cd-483b-8c2b-4636a0a78714 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:06.789633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1044/2099] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:07:06 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:07:06.793898 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31aaea9d-49b8-4723-b8a4-2182b8863226 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:06.804354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31aaea9d-49b8-4723-b8a4-2182b8863226 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:06.808929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57b2ad1-7fec-4d48-8014-61902f4f22d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:06.809620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57b2ad1-7fec-4d48-8014-61902f4f22d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:06.809880 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57b2ad1-7fec-4d48-8014-61902f4f22d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:06.810153 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57b2ad1-7fec-4d48-8014-61902f4f22d3 admin admin] PATH_INFO: `/v3/projects/f47f120d5fbd4b909d1550289b2256a5/users/8ea47508708a43038a3b2d04429aa06d/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:06.825035 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57b2ad1-7fec-4d48-8014-61902f4f22d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f47f120d5fbd4b909d1550289b2256a5, user_id=8ea47508708a43038a3b2d04429aa06d, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:06.827704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57b2ad1-7fec-4d48-8014-61902f4f22d3 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:06.920048 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1056/2100] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:07:06 2022] PUT /identity/v3/projects/f47f120d5fbd4b909d1550289b2256a5/users/8ea47508708a43038a3b2d04429aa06d/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:07:06.931375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab89faa-2a22-4a63-82d0-39cf6ccee86d None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:06.931660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab89faa-2a22-4a63-82d0-39cf6ccee86d None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:06.931932 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab89faa-2a22-4a63-82d0-39cf6ccee86d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:06.959668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-0ab89faa-2a22-4a63-82d0-39cf6ccee86d None None] MFA Rules not processed for user `8ea47508708a43038a3b2d04429aa06d`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:06.961008 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-0ab89faa-2a22-4a63-82d0-39cf6ccee86d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:07.026263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1045/2101] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:06 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:07.032489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57b2ad1-7fec-4d48-8014-61902f4f22d3 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:07.054575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:07.096673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57b2ad1-7fec-4d48-8014-61902f4f22d3 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:07.103390 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ebf596-ff8a-4c9d-bbdb-07f8d4702de4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:07.103993 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ebf596-ff8a-4c9d-bbdb-07f8d4702de4 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:07.104305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ebf596-ff8a-4c9d-bbdb-07f8d4702de4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:07.104585 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ebf596-ff8a-4c9d-bbdb-07f8d4702de4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:07.105267 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ebf596-ff8a-4c9d-bbdb-07f8d4702de4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:07.107370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ebf596-ff8a-4c9d-bbdb-07f8d4702de4 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:07.275466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1057/2102] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:07:07 2022] POST /identity/v3/projects => generated 381 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:07:07.279760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab89faa-2a22-4a63-82d0-39cf6ccee86d None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:07.290975 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab89faa-2a22-4a63-82d0-39cf6ccee86d None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:07.303129 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:07.350792 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892dcb73-99b2-4f13-9bb2-da2cd86cd58a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:07.351410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892dcb73-99b2-4f13-9bb2-da2cd86cd58a admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:07.351686 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892dcb73-99b2-4f13-9bb2-da2cd86cd58a admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:07.351958 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892dcb73-99b2-4f13-9bb2-da2cd86cd58a admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:07.352731 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892dcb73-99b2-4f13-9bb2-da2cd86cd58a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:07.354729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892dcb73-99b2-4f13-9bb2-da2cd86cd58a admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:07.462887 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1046/2103] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:07:07 2022] POST /identity/v3/users => generated 297 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:07:07.470077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ebf596-ff8a-4c9d-bbdb-07f8d4702de4 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:07.477299 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ebf596-ff8a-4c9d-bbdb-07f8d4702de4 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:07.481883 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab8608c-4459-408f-ad08-b032ceebeb9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:07.482479 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab8608c-4459-408f-ad08-b032ceebeb9a admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:07.482756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab8608c-4459-408f-ad08-b032ceebeb9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:07.483029 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab8608c-4459-408f-ad08-b032ceebeb9a admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:07.483666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab8608c-4459-408f-ad08-b032ceebeb9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:07.485574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab8608c-4459-408f-ad08-b032ceebeb9a admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:07.496742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1058/2104] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:07:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:07:07.500979 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892dcb73-99b2-4f13-9bb2-da2cd86cd58a admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:07.512919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892dcb73-99b2-4f13-9bb2-da2cd86cd58a admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:07.518067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc9c427-3c53-43e0-91d4-6220e916a273 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:07.518741 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc9c427-3c53-43e0-91d4-6220e916a273 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:07.520086 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc9c427-3c53-43e0-91d4-6220e916a273 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:07.520350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc9c427-3c53-43e0-91d4-6220e916a273 admin admin] PATH_INFO: `/v3/projects/604187a2ab3645bbacaa14dfa446c3f6/users/2de12963497f41028a2319f88a65ca89/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:07.535162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc9c427-3c53-43e0-91d4-6220e916a273 admin admin] RBAC: Authorizing `identity:create_grant(project_id=604187a2ab3645bbacaa14dfa446c3f6, user_id=2de12963497f41028a2319f88a65ca89, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:07.537764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc9c427-3c53-43e0-91d4-6220e916a273 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:07.610180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1047/2105] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:07:07 2022] PUT /identity/v3/projects/604187a2ab3645bbacaa14dfa446c3f6/users/2de12963497f41028a2319f88a65ca89/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:07:07.614982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab8608c-4459-408f-ad08-b032ceebeb9a admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:07.629714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab8608c-4459-408f-ad08-b032ceebeb9a admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:07.702427 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd21522a-a7aa-4c99-84fb-52d39f863330 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:07.703044 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd21522a-a7aa-4c99-84fb-52d39f863330 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:07.703324 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd21522a-a7aa-4c99-84fb-52d39f863330 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:07.703602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd21522a-a7aa-4c99-84fb-52d39f863330 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:07.704244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd21522a-a7aa-4c99-84fb-52d39f863330 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:07.705867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd21522a-a7aa-4c99-84fb-52d39f863330 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:07.716707 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1059/2106] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:07:07 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:07:07.720621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc9c427-3c53-43e0-91d4-6220e916a273 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:07.733621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc9c427-3c53-43e0-91d4-6220e916a273 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:07.740930 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9741dac4-0388-4834-a928-cba5c480a790 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:07.741618 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9741dac4-0388-4834-a928-cba5c480a790 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:07.741901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9741dac4-0388-4834-a928-cba5c480a790 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:07.742178 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9741dac4-0388-4834-a928-cba5c480a790 admin admin] PATH_INFO: `/v3/projects/604187a2ab3645bbacaa14dfa446c3f6/users/2de12963497f41028a2319f88a65ca89/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:07.744106 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9741dac4-0388-4834-a928-cba5c480a790 admin admin] RBAC: Authorizing `identity:create_grant(project_id=604187a2ab3645bbacaa14dfa446c3f6, user_id=2de12963497f41028a2319f88a65ca89, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:07.746961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9741dac4-0388-4834-a928-cba5c480a790 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:07.759157 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1048/2107] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:07:07 2022] PUT /identity/v3/projects/604187a2ab3645bbacaa14dfa446c3f6/users/2de12963497f41028a2319f88a65ca89/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:07:07.770026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0722df0-744b-4f1f-a796-f1944dc9996b None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:07.770312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0722df0-744b-4f1f-a796-f1944dc9996b None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:07.770587 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0722df0-744b-4f1f-a796-f1944dc9996b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:07.797837 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-b0722df0-744b-4f1f-a796-f1944dc9996b None None] MFA Rules not processed for user `2de12963497f41028a2319f88a65ca89`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:07.798981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b0722df0-744b-4f1f-a796-f1944dc9996b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:07.873159 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1060/2108] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:07 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:07.884923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9741dac4-0388-4834-a928-cba5c480a790 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:07.886653 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-9741dac4-0388-4834-a928-cba5c480a790 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:07.948760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9741dac4-0388-4834-a928-cba5c480a790 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:07.952628 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53294c3-de44-4464-a3f4-59c5c81f7e77 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:07.953285 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53294c3-de44-4464-a3f4-59c5c81f7e77 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:07.953565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53294c3-de44-4464-a3f4-59c5c81f7e77 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:07.953836 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53294c3-de44-4464-a3f4-59c5c81f7e77 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:07.962976 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53294c3-de44-4464-a3f4-59c5c81f7e77 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:07.964394 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53294c3-de44-4464-a3f4-59c5c81f7e77 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:07.973758 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1049/2109] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:07:07 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:07:08.501569 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0722df0-744b-4f1f-a796-f1944dc9996b None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:08.522383 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:08.571817 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0722df0-744b-4f1f-a796-f1944dc9996b None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:08.576286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bc63da-7312-494a-b96e-759a7c215595 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:08.576915 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bc63da-7312-494a-b96e-759a7c215595 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:08.577234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bc63da-7312-494a-b96e-759a7c215595 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:08.577510 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bc63da-7312-494a-b96e-759a7c215595 admin admin] PATH_INFO: `/v3/users/8ea47508708a43038a3b2d04429aa06d` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:08.578626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bc63da-7312-494a-b96e-759a7c215595 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ea47508708a43038a3b2d04429aa06d)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:08.580596 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bc63da-7312-494a-b96e-759a7c215595 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:08.756070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-91bc63da-7312-494a-b96e-759a7c215595 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ea47508708a43038a3b2d04429aa06d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:08.826659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-91bc63da-7312-494a-b96e-759a7c215595 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ea47508708a43038a3b2d04429aa06d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:08.843727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-91bc63da-7312-494a-b96e-759a7c215595 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ea47508708a43038a3b2d04429aa06d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:08.844478 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-91bc63da-7312-494a-b96e-759a7c215595 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ea47508708a43038a3b2d04429aa06d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:08.852593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-91bc63da-7312-494a-b96e-759a7c215595 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ea47508708a43038a3b2d04429aa06d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:08.862534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1061/2110] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:07:08 2022] DELETE /identity/v3/users/8ea47508708a43038a3b2d04429aa06d => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:07:08.871476 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be3e536-0363-489f-9e93-543f4de7b911 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:08.871981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be3e536-0363-489f-9e93-543f4de7b911 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:08.872077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be3e536-0363-489f-9e93-543f4de7b911 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:08.892572 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-8be3e536-0363-489f-9e93-543f4de7b911 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:08.900157 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8be3e536-0363-489f-9e93-543f4de7b911 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:08.955509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1050/2111] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:08.966433 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bc63da-7312-494a-b96e-759a7c215595 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:08.967847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-91bc63da-7312-494a-b96e-759a7c215595 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:09.030939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bc63da-7312-494a-b96e-759a7c215595 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:09.034696 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6944bdfd-8e1a-46ff-b2fb-95cd497d9a3e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:09.037883 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6944bdfd-8e1a-46ff-b2fb-95cd497d9a3e service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:09.037883 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6944bdfd-8e1a-46ff-b2fb-95cd497d9a3e service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:09.037883 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6944bdfd-8e1a-46ff-b2fb-95cd497d9a3e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:09.045220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6944bdfd-8e1a-46ff-b2fb-95cd497d9a3e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:09.046661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6944bdfd-8e1a-46ff-b2fb-95cd497d9a3e service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:09.056632 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1062/2112] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:07:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:07:09.469442 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be3e536-0363-489f-9e93-543f4de7b911 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:09.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8be3e536-0363-489f-9e93-543f4de7b911 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:09.483214 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be3e536-0363-489f-9e93-543f4de7b911 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:09.488082 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130a8a4b-6c8a-4cde-8d49-689d7174d535 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:09.489416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130a8a4b-6c8a-4cde-8d49-689d7174d535 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:09.489657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130a8a4b-6c8a-4cde-8d49-689d7174d535 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:09.489891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130a8a4b-6c8a-4cde-8d49-689d7174d535 admin admin] PATH_INFO: `/v3/projects/f47f120d5fbd4b909d1550289b2256a5` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:09.491148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130a8a4b-6c8a-4cde-8d49-689d7174d535 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f47f120d5fbd4b909d1550289b2256a5)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:09.493397 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130a8a4b-6c8a-4cde-8d49-689d7174d535 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:09.503608 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:09.626470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-130a8a4b-6c8a-4cde-8d49-689d7174d535 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f47f120d5fbd4b909d1550289b2256a5', 'request_id': 'req-130a8a4b-6c8a-4cde-8d49-689d7174d535'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:09.633197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-130a8a4b-6c8a-4cde-8d49-689d7174d535 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f47f120d5fbd4b909d1550289b2256a5', 'request_id': 'req-130a8a4b-6c8a-4cde-8d49-689d7174d535'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:09.718385 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-130a8a4b-6c8a-4cde-8d49-689d7174d535 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f47f120d5fbd4b909d1550289b2256a5', 'request_id': 'req-130a8a4b-6c8a-4cde-8d49-689d7174d535'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:09.732195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-130a8a4b-6c8a-4cde-8d49-689d7174d535 admin admin] The token cache is being invalidate because project f47f120d5fbd4b909d1550289b2256a5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:07:09.738174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-130a8a4b-6c8a-4cde-8d49-689d7174d535 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:09.738174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-130a8a4b-6c8a-4cde-8d49-689d7174d535 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:09.738174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1051/2113] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:07:09 2022] DELETE /identity/v3/projects/f47f120d5fbd4b909d1550289b2256a5 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:07:09.740178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6944bdfd-8e1a-46ff-b2fb-95cd497d9a3e service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:09.762207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6944bdfd-8e1a-46ff-b2fb-95cd497d9a3e service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:09.803387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:09.825524 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea5a981-780e-4954-9939-e29456bb06d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:09.826152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea5a981-780e-4954-9939-e29456bb06d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:09.830452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea5a981-780e-4954-9939-e29456bb06d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:09.830452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea5a981-780e-4954-9939-e29456bb06d3 admin admin] PATH_INFO: `/v3/users/2de12963497f41028a2319f88a65ca89` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:09.830452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea5a981-780e-4954-9939-e29456bb06d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2de12963497f41028a2319f88a65ca89)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:09.830452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea5a981-780e-4954-9939-e29456bb06d3 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:10.028282 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-aea5a981-780e-4954-9939-e29456bb06d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2de12963497f41028a2319f88a65ca89'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:10.091628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-aea5a981-780e-4954-9939-e29456bb06d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2de12963497f41028a2319f88a65ca89'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:10.111764 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-aea5a981-780e-4954-9939-e29456bb06d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2de12963497f41028a2319f88a65ca89'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:10.118453 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-aea5a981-780e-4954-9939-e29456bb06d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2de12963497f41028a2319f88a65ca89'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:10.131349 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-aea5a981-780e-4954-9939-e29456bb06d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2de12963497f41028a2319f88a65ca89'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:10.147721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1063/2114] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:07:09 2022] DELETE /identity/v3/users/2de12963497f41028a2319f88a65ca89 => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:07:10.731355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130a8a4b-6c8a-4cde-8d49-689d7174d535 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:10.733057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-130a8a4b-6c8a-4cde-8d49-689d7174d535 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:10.746454 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130a8a4b-6c8a-4cde-8d49-689d7174d535 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:10.811305 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66f1820-41bc-4fc2-9242-3994b47ba4ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:10.812045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66f1820-41bc-4fc2-9242-3994b47ba4ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:10.812396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66f1820-41bc-4fc2-9242-3994b47ba4ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:10.812732 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66f1820-41bc-4fc2-9242-3994b47ba4ff admin admin] PATH_INFO: `/v3/projects/604187a2ab3645bbacaa14dfa446c3f6` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:10.814011 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c66f1820-41bc-4fc2-9242-3994b47ba4ff admin admin] RBAC: Authorizing `identity:delete_project(project_id=604187a2ab3645bbacaa14dfa446c3f6)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:10.816362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c66f1820-41bc-4fc2-9242-3994b47ba4ff admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:10.938622 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c66f1820-41bc-4fc2-9242-3994b47ba4ff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '604187a2ab3645bbacaa14dfa446c3f6', 'request_id': 'req-c66f1820-41bc-4fc2-9242-3994b47ba4ff'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:10.945597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c66f1820-41bc-4fc2-9242-3994b47ba4ff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '604187a2ab3645bbacaa14dfa446c3f6', 'request_id': 'req-c66f1820-41bc-4fc2-9242-3994b47ba4ff'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:10.998090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c66f1820-41bc-4fc2-9242-3994b47ba4ff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '604187a2ab3645bbacaa14dfa446c3f6', 'request_id': 'req-c66f1820-41bc-4fc2-9242-3994b47ba4ff'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:11.010494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c66f1820-41bc-4fc2-9242-3994b47ba4ff admin admin] The token cache is being invalidate because project 604187a2ab3645bbacaa14dfa446c3f6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:07:11.010783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c66f1820-41bc-4fc2-9242-3994b47ba4ff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:11.011414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c66f1820-41bc-4fc2-9242-3994b47ba4ff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:11.012919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1052/2115] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:07:10 2022] DELETE /identity/v3/projects/604187a2ab3645bbacaa14dfa446c3f6 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:07:11.018667 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcedb20-297b-4544-ba1c-9a7b4731097b None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:11.019134 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcedb20-297b-4544-ba1c-9a7b4731097b None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:11.019292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcedb20-297b-4544-ba1c-9a7b4731097b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:11.039355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-fdcedb20-297b-4544-ba1c-9a7b4731097b None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:11.044199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-fdcedb20-297b-4544-ba1c-9a7b4731097b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:11.100931 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1064/2116] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:11.110246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555125fc-7bcb-47b8-a45c-63e039bb039e None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:11.110553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555125fc-7bcb-47b8-a45c-63e039bb039e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:11.110829 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555125fc-7bcb-47b8-a45c-63e039bb039e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:11.132184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-555125fc-7bcb-47b8-a45c-63e039bb039e None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:11.133476 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-555125fc-7bcb-47b8-a45c-63e039bb039e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:11.141724 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1053/2117] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:11.145965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcedb20-297b-4544-ba1c-9a7b4731097b None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:11.157106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcedb20-297b-4544-ba1c-9a7b4731097b None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:11.163677 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da314261-b453-4d3a-a130-c966216c32b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:11.164567 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da314261-b453-4d3a-a130-c966216c32b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:11.164910 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da314261-b453-4d3a-a130-c966216c32b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:11.165409 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da314261-b453-4d3a-a130-c966216c32b8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:11.166135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da314261-b453-4d3a-a130-c966216c32b8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:07:11.166504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da314261-b453-4d3a-a130-c966216c32b8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:11.169197 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da314261-b453-4d3a-a130-c966216c32b8 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:11.197944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1065/2118] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:07:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:07:11.213299 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555125fc-7bcb-47b8-a45c-63e039bb039e None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:11.213299 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555125fc-7bcb-47b8-a45c-63e039bb039e None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:11.216589 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea10ee4-6c50-44c5-85ea-ec6012a29f4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:11.217308 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea10ee4-6c50-44c5-85ea-ec6012a29f4f admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:11.217662 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea10ee4-6c50-44c5-85ea-ec6012a29f4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:11.218008 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea10ee4-6c50-44c5-85ea-ec6012a29f4f admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:11.218727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea10ee4-6c50-44c5-85ea-ec6012a29f4f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:11.220627 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea10ee4-6c50-44c5-85ea-ec6012a29f4f admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:11.314526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:11.326370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1054/2119] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:07:11 2022] POST /identity/v3/projects => generated 385 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:07:11.330585 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da314261-b453-4d3a-a130-c966216c32b8 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:11.340805 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da314261-b453-4d3a-a130-c966216c32b8 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:11.382745 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:11.397312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc500f8f-ce93-4162-8ad4-0b279629f0f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:11.397955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc500f8f-ce93-4162-8ad4-0b279629f0f2 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:11.398244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc500f8f-ce93-4162-8ad4-0b279629f0f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:11.398520 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc500f8f-ce93-4162-8ad4-0b279629f0f2 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:11.399157 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc500f8f-ce93-4162-8ad4-0b279629f0f2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:11.401088 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc500f8f-ce93-4162-8ad4-0b279629f0f2 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:11.497475 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1066/2120] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:07:11 2022] POST /identity/v3/users => generated 293 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:07:11.501922 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea10ee4-6c50-44c5-85ea-ec6012a29f4f admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:11.512510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea10ee4-6c50-44c5-85ea-ec6012a29f4f admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:11.517267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191efbb4-26b6-4d46-a1ad-8aa89c0b6883 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:11.517890 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191efbb4-26b6-4d46-a1ad-8aa89c0b6883 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:11.518174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191efbb4-26b6-4d46-a1ad-8aa89c0b6883 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:11.518464 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191efbb4-26b6-4d46-a1ad-8aa89c0b6883 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:11.519111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191efbb4-26b6-4d46-a1ad-8aa89c0b6883 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:11.520729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191efbb4-26b6-4d46-a1ad-8aa89c0b6883 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:11.530874 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1055/2121] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:07:11 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:07:11.534922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc500f8f-ce93-4162-8ad4-0b279629f0f2 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:11.545373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc500f8f-ce93-4162-8ad4-0b279629f0f2 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:11.550411 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8719da64-a417-40b7-a2dd-d216c45c51c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:11.551012 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8719da64-a417-40b7-a2dd-d216c45c51c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:11.551888 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8719da64-a417-40b7-a2dd-d216c45c51c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:11.552184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8719da64-a417-40b7-a2dd-d216c45c51c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:11.552865 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8719da64-a417-40b7-a2dd-d216c45c51c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:11.554557 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8719da64-a417-40b7-a2dd-d216c45c51c4 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:11.565059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1067/2122] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:07:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:07:11.569215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191efbb4-26b6-4d46-a1ad-8aa89c0b6883 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:11.579283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191efbb4-26b6-4d46-a1ad-8aa89c0b6883 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:11.584761 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eabdddf-8018-4d7a-bff9-f8cbcffa9029 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:11.585441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eabdddf-8018-4d7a-bff9-f8cbcffa9029 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:11.585741 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eabdddf-8018-4d7a-bff9-f8cbcffa9029 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:11.585996 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eabdddf-8018-4d7a-bff9-f8cbcffa9029 admin admin] PATH_INFO: `/v3/projects/d57426a3bc074d01b88e5fb9cb7fa71e/users/00d9424d6dde4a0b87cc1ce9e7a93ec0/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:11.602361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eabdddf-8018-4d7a-bff9-f8cbcffa9029 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d57426a3bc074d01b88e5fb9cb7fa71e, user_id=00d9424d6dde4a0b87cc1ce9e7a93ec0, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:11.605244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eabdddf-8018-4d7a-bff9-f8cbcffa9029 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:11.697930 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1056/2123] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:07:11 2022] PUT /identity/v3/projects/d57426a3bc074d01b88e5fb9cb7fa71e/users/00d9424d6dde4a0b87cc1ce9e7a93ec0/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:07:11.708972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0005eff1-4e4e-4d59-b927-eb4e8da6a358 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:11.709352 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0005eff1-4e4e-4d59-b927-eb4e8da6a358 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:11.709647 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0005eff1-4e4e-4d59-b927-eb4e8da6a358 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:11.740961 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-0005eff1-4e4e-4d59-b927-eb4e8da6a358 None None] MFA Rules not processed for user `00d9424d6dde4a0b87cc1ce9e7a93ec0`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:11.743106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0005eff1-4e4e-4d59-b927-eb4e8da6a358 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:11.823710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1068/2124] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:11 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:11.835572 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eabdddf-8018-4d7a-bff9-f8cbcffa9029 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:11.837822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8eabdddf-8018-4d7a-bff9-f8cbcffa9029 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:11.897692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eabdddf-8018-4d7a-bff9-f8cbcffa9029 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:11.901729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55df58a5-166b-4b58-bf4f-1bd7f7b2053c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:11.902278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55df58a5-166b-4b58-bf4f-1bd7f7b2053c service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:11.902586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55df58a5-166b-4b58-bf4f-1bd7f7b2053c service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:11.902869 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55df58a5-166b-4b58-bf4f-1bd7f7b2053c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:11.911877 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55df58a5-166b-4b58-bf4f-1bd7f7b2053c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:11.913321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55df58a5-166b-4b58-bf4f-1bd7f7b2053c service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:11.922318 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1057/2125] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:07:11 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:07:20.033841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0005eff1-4e4e-4d59-b927-eb4e8da6a358 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:20.035321 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0005eff1-4e4e-4d59-b927-eb4e8da6a358 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:20.103151 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0005eff1-4e4e-4d59-b927-eb4e8da6a358 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:20.108322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84edaa7-cc25-43bc-bab9-0e1e5bc138d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:20.109487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84edaa7-cc25-43bc-bab9-0e1e5bc138d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:20.110368 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84edaa7-cc25-43bc-bab9-0e1e5bc138d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:20.110654 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84edaa7-cc25-43bc-bab9-0e1e5bc138d0 admin admin] PATH_INFO: `/v3/users/b234abcd2cf5460cb23edc68489d4592` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:20.111829 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84edaa7-cc25-43bc-bab9-0e1e5bc138d0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b234abcd2cf5460cb23edc68489d4592)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:20.114623 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84edaa7-cc25-43bc-bab9-0e1e5bc138d0 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:20.352757 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-c84edaa7-cc25-43bc-bab9-0e1e5bc138d0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b234abcd2cf5460cb23edc68489d4592'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:20.456307 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-c84edaa7-cc25-43bc-bab9-0e1e5bc138d0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b234abcd2cf5460cb23edc68489d4592'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:20.458202 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:20.476991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-c84edaa7-cc25-43bc-bab9-0e1e5bc138d0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b234abcd2cf5460cb23edc68489d4592'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:20.477940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-c84edaa7-cc25-43bc-bab9-0e1e5bc138d0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b234abcd2cf5460cb23edc68489d4592'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:20.485224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-c84edaa7-cc25-43bc-bab9-0e1e5bc138d0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b234abcd2cf5460cb23edc68489d4592'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:20.491961 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1069/2126] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:07:20 2022] DELETE /identity/v3/users/b234abcd2cf5460cb23edc68489d4592 => generated 0 bytes in 459 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:07:20.496483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6baa744-772c-4f9c-9881-7d4b39a9487e None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:20.496934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6baa744-772c-4f9c-9881-7d4b39a9487e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:20.497025 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6baa744-772c-4f9c-9881-7d4b39a9487e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:20.515881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-c6baa744-772c-4f9c-9881-7d4b39a9487e None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:20.517146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-c6baa744-772c-4f9c-9881-7d4b39a9487e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:20.578227 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1058/2127] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:20.588545 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84edaa7-cc25-43bc-bab9-0e1e5bc138d0 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:20.590008 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c84edaa7-cc25-43bc-bab9-0e1e5bc138d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:20.648471 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84edaa7-cc25-43bc-bab9-0e1e5bc138d0 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:20.655653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad73c66-6237-4385-9f6e-e8c623471a14 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:20.655653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad73c66-6237-4385-9f6e-e8c623471a14 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:20.655653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad73c66-6237-4385-9f6e-e8c623471a14 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:20.655653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad73c66-6237-4385-9f6e-e8c623471a14 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:20.664477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad73c66-6237-4385-9f6e-e8c623471a14 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:20.665962 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad73c66-6237-4385-9f6e-e8c623471a14 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:20.677311 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1070/2128] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:07:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:07:21.126357 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6baa744-772c-4f9c-9881-7d4b39a9487e None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:21.127795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-c6baa744-772c-4f9c-9881-7d4b39a9487e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:21.140927 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6baa744-772c-4f9c-9881-7d4b39a9487e None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:21.147963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015d81e9-2017-4130-a8c5-873c855517b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:21.148610 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015d81e9-2017-4130-a8c5-873c855517b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:21.148857 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015d81e9-2017-4130-a8c5-873c855517b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:21.149463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015d81e9-2017-4130-a8c5-873c855517b0 admin admin] PATH_INFO: `/v3/projects/2578bf50dc5948f6a7a9373cb020282b` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:21.158348 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015d81e9-2017-4130-a8c5-873c855517b0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2578bf50dc5948f6a7a9373cb020282b)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:21.164169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-015d81e9-2017-4130-a8c5-873c855517b0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:21.297934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-015d81e9-2017-4130-a8c5-873c855517b0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2578bf50dc5948f6a7a9373cb020282b', 'request_id': 'req-015d81e9-2017-4130-a8c5-873c855517b0'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:21.304811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-015d81e9-2017-4130-a8c5-873c855517b0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2578bf50dc5948f6a7a9373cb020282b', 'request_id': 'req-015d81e9-2017-4130-a8c5-873c855517b0'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:21.307602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:21.416349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-015d81e9-2017-4130-a8c5-873c855517b0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2578bf50dc5948f6a7a9373cb020282b', 'request_id': 'req-015d81e9-2017-4130-a8c5-873c855517b0'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:21.435963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-015d81e9-2017-4130-a8c5-873c855517b0 admin admin] The token cache is being invalidate because project 2578bf50dc5948f6a7a9373cb020282b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:07:21.436407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-015d81e9-2017-4130-a8c5-873c855517b0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:21.437232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-015d81e9-2017-4130-a8c5-873c855517b0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:21.440088 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1059/2129] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:07:21 2022] DELETE /identity/v3/projects/2578bf50dc5948f6a7a9373cb020282b => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:07:21.445980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae877bf-b88b-4f37-94a9-72d413bbb7e4 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:21.446473 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae877bf-b88b-4f37-94a9-72d413bbb7e4 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:21.446567 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae877bf-b88b-4f37-94a9-72d413bbb7e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:21.466738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-aae877bf-b88b-4f37-94a9-72d413bbb7e4 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:21.467972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-aae877bf-b88b-4f37-94a9-72d413bbb7e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:21.537224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1071/2130] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:21.545459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa9dc57-2d9c-45c6-a3cd-6901a0e345d0 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:21.545947 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa9dc57-2d9c-45c6-a3cd-6901a0e345d0 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:21.546140 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa9dc57-2d9c-45c6-a3cd-6901a0e345d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:21.573902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-4fa9dc57-2d9c-45c6-a3cd-6901a0e345d0 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:21.575614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-4fa9dc57-2d9c-45c6-a3cd-6901a0e345d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:21.586054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1060/2131] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:21.590546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae877bf-b88b-4f37-94a9-72d413bbb7e4 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:21.600895 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae877bf-b88b-4f37-94a9-72d413bbb7e4 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:21.605541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c586b22-0d8c-4f73-a3b2-b494ffdfbba2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:21.606141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c586b22-0d8c-4f73-a3b2-b494ffdfbba2 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:21.606419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c586b22-0d8c-4f73-a3b2-b494ffdfbba2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:21.606690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c586b22-0d8c-4f73-a3b2-b494ffdfbba2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:21.607344 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c586b22-0d8c-4f73-a3b2-b494ffdfbba2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:07:21.607646 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c586b22-0d8c-4f73-a3b2-b494ffdfbba2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:21.609274 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c586b22-0d8c-4f73-a3b2-b494ffdfbba2 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:21.621736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1072/2132] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:07:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:07:21.626861 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa9dc57-2d9c-45c6-a3cd-6901a0e345d0 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:21.636512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa9dc57-2d9c-45c6-a3cd-6901a0e345d0 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:21.641895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d9f542-18f2-4f4f-b37a-b325895eaea1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:21.642491 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d9f542-18f2-4f4f-b37a-b325895eaea1 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:21.642957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d9f542-18f2-4f4f-b37a-b325895eaea1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:21.643070 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d9f542-18f2-4f4f-b37a-b325895eaea1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:21.643694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d9f542-18f2-4f4f-b37a-b325895eaea1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:21.646473 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d9f542-18f2-4f4f-b37a-b325895eaea1 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:21.764992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1061/2133] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:07:21 2022] POST /identity/v3/projects => generated 365 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:07:21.769331 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c586b22-0d8c-4f73-a3b2-b494ffdfbba2 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:21.780701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c586b22-0d8c-4f73-a3b2-b494ffdfbba2 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:21.849678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24c1158-4bda-4c19-88a7-1ce086bb630c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:21.850290 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c1158-4bda-4c19-88a7-1ce086bb630c admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:21.850607 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c1158-4bda-4c19-88a7-1ce086bb630c admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:21.850831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c1158-4bda-4c19-88a7-1ce086bb630c admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:21.851504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24c1158-4bda-4c19-88a7-1ce086bb630c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:21.853613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24c1158-4bda-4c19-88a7-1ce086bb630c admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:21.863729 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:21.943519 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1073/2134] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:07:21 2022] POST /identity/v3/users => generated 283 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:07:21.944484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d9f542-18f2-4f4f-b37a-b325895eaea1 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:21.954779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d9f542-18f2-4f4f-b37a-b325895eaea1 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:21.961048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c24388-4997-4b5a-827c-520ff5c3a9a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:21.961679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c24388-4997-4b5a-827c-520ff5c3a9a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:21.961956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c24388-4997-4b5a-827c-520ff5c3a9a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:21.962229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c24388-4997-4b5a-827c-520ff5c3a9a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:21.962913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c24388-4997-4b5a-827c-520ff5c3a9a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:21.963498 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24c1158-4bda-4c19-88a7-1ce086bb630c admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:21.964500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c24388-4997-4b5a-827c-520ff5c3a9a0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:21.965326 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b24c1158-4bda-4c19-88a7-1ce086bb630c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:21.975014 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1062/2135] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:07:21 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:07:21.977771 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24c1158-4bda-4c19-88a7-1ce086bb630c admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:21.978839 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c24388-4997-4b5a-827c-520ff5c3a9a0 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:21.982659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b50a56-e8a1-41da-8962-ace4404bacec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:21.983478 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b50a56-e8a1-41da-8962-ace4404bacec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:21.983580 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b50a56-e8a1-41da-8962-ace4404bacec admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:21.983880 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b50a56-e8a1-41da-8962-ace4404bacec admin admin] PATH_INFO: `/v3/users/00d9424d6dde4a0b87cc1ce9e7a93ec0` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:21.985317 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b50a56-e8a1-41da-8962-ace4404bacec admin admin] RBAC: Authorizing `identity:delete_user(user_id=00d9424d6dde4a0b87cc1ce9e7a93ec0)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:21.987281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b50a56-e8a1-41da-8962-ace4404bacec admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:21.988841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c24388-4997-4b5a-827c-520ff5c3a9a0 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:21.993439 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e9f268-1455-43d3-8c24-4b18ed6ea401 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:21.994027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e9f268-1455-43d3-8c24-4b18ed6ea401 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:21.994302 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e9f268-1455-43d3-8c24-4b18ed6ea401 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:21.994572 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e9f268-1455-43d3-8c24-4b18ed6ea401 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:21.995210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e9f268-1455-43d3-8c24-4b18ed6ea401 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:21.996761 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e9f268-1455-43d3-8c24-4b18ed6ea401 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:22.007499 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1063/2136] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:07:21 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:07:22.012222 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e9f268-1455-43d3-8c24-4b18ed6ea401 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:22.022496 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e9f268-1455-43d3-8c24-4b18ed6ea401 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:22.027060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b914b19c-fe88-475d-a0a9-a1803eaf8d58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:22.027701 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b914b19c-fe88-475d-a0a9-a1803eaf8d58 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:22.027978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b914b19c-fe88-475d-a0a9-a1803eaf8d58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:22.029362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b914b19c-fe88-475d-a0a9-a1803eaf8d58 admin admin] PATH_INFO: `/v3/projects/a0f43a31c7e549eeb636fcce144cb3bc/users/d79e61978ff5400683c838d77a6ebb7e/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:22.043630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b914b19c-fe88-475d-a0a9-a1803eaf8d58 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0f43a31c7e549eeb636fcce144cb3bc, user_id=d79e61978ff5400683c838d77a6ebb7e, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:22.046313 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b914b19c-fe88-475d-a0a9-a1803eaf8d58 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:22.135808 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1064/2137] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:07:22 2022] PUT /identity/v3/projects/a0f43a31c7e549eeb636fcce144cb3bc/users/d79e61978ff5400683c838d77a6ebb7e/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:07:22.145888 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64572f29-887d-459f-ac4a-415eeedc4ed6 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:22.146174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64572f29-887d-459f-ac4a-415eeedc4ed6 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:22.146459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64572f29-887d-459f-ac4a-415eeedc4ed6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:22.172936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-64572f29-887d-459f-ac4a-415eeedc4ed6 None None] MFA Rules not processed for user `d79e61978ff5400683c838d77a6ebb7e`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:22.174175 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-64572f29-887d-459f-ac4a-415eeedc4ed6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:22.201478 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-a1b50a56-e8a1-41da-8962-ace4404bacec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '00d9424d6dde4a0b87cc1ce9e7a93ec0'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:22.245162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1065/2138] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:22 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:07:22.269382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64572f29-887d-459f-ac4a-415eeedc4ed6 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:22.269382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-64572f29-887d-459f-ac4a-415eeedc4ed6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:22.317999 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64572f29-887d-459f-ac4a-415eeedc4ed6 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:22.322225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d258222-cc7c-42ba-8abb-ddcc14708c23 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:22.322341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d258222-cc7c-42ba-8abb-ddcc14708c23 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:22.322584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d258222-cc7c-42ba-8abb-ddcc14708c23 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:22.322877 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d258222-cc7c-42ba-8abb-ddcc14708c23 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:22.330936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d258222-cc7c-42ba-8abb-ddcc14708c23 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:22.332394 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d258222-cc7c-42ba-8abb-ddcc14708c23 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:22.341026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1066/2139] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:07:22 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:07:22.407756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-a1b50a56-e8a1-41da-8962-ace4404bacec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '00d9424d6dde4a0b87cc1ce9e7a93ec0'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:22.424033 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-a1b50a56-e8a1-41da-8962-ace4404bacec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '00d9424d6dde4a0b87cc1ce9e7a93ec0'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:22.430906 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-a1b50a56-e8a1-41da-8962-ace4404bacec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '00d9424d6dde4a0b87cc1ce9e7a93ec0'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:22.441218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-a1b50a56-e8a1-41da-8962-ace4404bacec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '00d9424d6dde4a0b87cc1ce9e7a93ec0'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:22.454985 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a0f3b5-6d2a-4a6e-b359-12d750ea22c7 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:22.455285 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a0f3b5-6d2a-4a6e-b359-12d750ea22c7 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:22.455546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a0f3b5-6d2a-4a6e-b359-12d750ea22c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:22.460418 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1074/2140] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:07:21 2022] DELETE /identity/v3/users/00d9424d6dde4a0b87cc1ce9e7a93ec0 => generated 0 bytes in 487 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:07:22.480144 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:DELETE=1,keystone:SELECT=97,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:22.492095 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-f8a0f3b5-6d2a-4a6e-b359-12d750ea22c7 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:22.493723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f8a0f3b5-6d2a-4a6e-b359-12d750ea22c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:22.581450 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1067/2141] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:22.591932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b50a56-e8a1-41da-8962-ace4404bacec admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:22.593337 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a1b50a56-e8a1-41da-8962-ace4404bacec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:22.656957 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b50a56-e8a1-41da-8962-ace4404bacec admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:22.660936 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2e3132-abea-4b4b-a3e4-b85fc506651f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:22.661629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2e3132-abea-4b4b-a3e4-b85fc506651f service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:22.661910 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2e3132-abea-4b4b-a3e4-b85fc506651f service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:22.662183 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2e3132-abea-4b4b-a3e4-b85fc506651f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:22.672098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2e3132-abea-4b4b-a3e4-b85fc506651f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:22.673581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2e3132-abea-4b4b-a3e4-b85fc506651f service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:22.683858 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1075/2142] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:07:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:07:23.042532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a0f3b5-6d2a-4a6e-b359-12d750ea22c7 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:23.043950 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f8a0f3b5-6d2a-4a6e-b359-12d750ea22c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:23.068042 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a0f3b5-6d2a-4a6e-b359-12d750ea22c7 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:23.086345 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa432dc-fba3-4bcc-8b02-89eda9a41ed4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:23.086345 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa432dc-fba3-4bcc-8b02-89eda9a41ed4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:23.086345 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa432dc-fba3-4bcc-8b02-89eda9a41ed4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:23.086345 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa432dc-fba3-4bcc-8b02-89eda9a41ed4 admin admin] PATH_INFO: `/v3/projects/d57426a3bc074d01b88e5fb9cb7fa71e` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:23.086783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa432dc-fba3-4bcc-8b02-89eda9a41ed4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d57426a3bc074d01b88e5fb9cb7fa71e)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:23.087306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa432dc-fba3-4bcc-8b02-89eda9a41ed4 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:23.225091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-bfa432dc-fba3-4bcc-8b02-89eda9a41ed4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd57426a3bc074d01b88e5fb9cb7fa71e', 'request_id': 'req-bfa432dc-fba3-4bcc-8b02-89eda9a41ed4'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:23.230933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-bfa432dc-fba3-4bcc-8b02-89eda9a41ed4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd57426a3bc074d01b88e5fb9cb7fa71e', 'request_id': 'req-bfa432dc-fba3-4bcc-8b02-89eda9a41ed4'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:23.297208 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-bfa432dc-fba3-4bcc-8b02-89eda9a41ed4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd57426a3bc074d01b88e5fb9cb7fa71e', 'request_id': 'req-bfa432dc-fba3-4bcc-8b02-89eda9a41ed4'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:23.310574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-bfa432dc-fba3-4bcc-8b02-89eda9a41ed4 admin admin] The token cache is being invalidate because project d57426a3bc074d01b88e5fb9cb7fa71e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:07:23.310853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-bfa432dc-fba3-4bcc-8b02-89eda9a41ed4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:23.311522 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-bfa432dc-fba3-4bcc-8b02-89eda9a41ed4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:07:23.313335 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1068/2143] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:07:23 2022] DELETE /identity/v3/projects/d57426a3bc074d01b88e5fb9cb7fa71e => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:07:23.318074 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7a1702-554d-4c0e-a86d-93674e26f067 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:23.318420 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7a1702-554d-4c0e-a86d-93674e26f067 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:23.318704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7a1702-554d-4c0e-a86d-93674e26f067 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:23.340012 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-0a7a1702-554d-4c0e-a86d-93674e26f067 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:23.341351 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0a7a1702-554d-4c0e-a86d-93674e26f067 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:23.404216 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1076/2144] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:23.412595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9299d637-060c-488a-b2ac-1b9ef849f373 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:23.412885 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9299d637-060c-488a-b2ac-1b9ef849f373 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:23.413289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9299d637-060c-488a-b2ac-1b9ef849f373 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:23.435119 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-9299d637-060c-488a-b2ac-1b9ef849f373 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:23.436453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-9299d637-060c-488a-b2ac-1b9ef849f373 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:23.444954 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1069/2145] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:23.449132 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7a1702-554d-4c0e-a86d-93674e26f067 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:23.458769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7a1702-554d-4c0e-a86d-93674e26f067 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:23.463370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067702c7-c149-4629-b9c3-91736c208c1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:23.463994 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067702c7-c149-4629-b9c3-91736c208c1c admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:23.464234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067702c7-c149-4629-b9c3-91736c208c1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:23.464513 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067702c7-c149-4629-b9c3-91736c208c1c admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:23.465179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067702c7-c149-4629-b9c3-91736c208c1c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:07:23.465483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067702c7-c149-4629-b9c3-91736c208c1c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:23.467077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067702c7-c149-4629-b9c3-91736c208c1c admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:23.468688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:INSERT=3,keystone:SELECT=87,keystone:DELETE=10 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:23.479819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1077/2146] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:07:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:07:23.484143 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9299d637-060c-488a-b2ac-1b9ef849f373 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:23.494915 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9299d637-060c-488a-b2ac-1b9ef849f373 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:23.499713 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35922281-f173-45d7-9fa8-c6b41a182e50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:23.500304 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35922281-f173-45d7-9fa8-c6b41a182e50 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:23.500590 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35922281-f173-45d7-9fa8-c6b41a182e50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:23.501034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35922281-f173-45d7-9fa8-c6b41a182e50 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:23.501718 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35922281-f173-45d7-9fa8-c6b41a182e50 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:23.503616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35922281-f173-45d7-9fa8-c6b41a182e50 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:23.601026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1070/2147] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:07:23 2022] POST /identity/v3/projects => generated 373 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:07:23.606358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067702c7-c149-4629-b9c3-91736c208c1c admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:23.617499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067702c7-c149-4629-b9c3-91736c208c1c admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:23.677012 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308fb926-99d8-4299-a0ab-871669002a07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:23.677811 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308fb926-99d8-4299-a0ab-871669002a07 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:23.678179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308fb926-99d8-4299-a0ab-871669002a07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:23.678456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308fb926-99d8-4299-a0ab-871669002a07 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:23.679305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308fb926-99d8-4299-a0ab-871669002a07 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:23.681359 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308fb926-99d8-4299-a0ab-871669002a07 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:23.753097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1078/2148] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:07:23 2022] POST /identity/v3/users => generated 287 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:07:23.757177 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35922281-f173-45d7-9fa8-c6b41a182e50 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:23.767329 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35922281-f173-45d7-9fa8-c6b41a182e50 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:23.772614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db634d3d-13e0-4c52-b69e-30ff505d9599 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:23.773248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db634d3d-13e0-4c52-b69e-30ff505d9599 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:23.773523 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db634d3d-13e0-4c52-b69e-30ff505d9599 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:23.773818 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db634d3d-13e0-4c52-b69e-30ff505d9599 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:23.774457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db634d3d-13e0-4c52-b69e-30ff505d9599 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:23.776042 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db634d3d-13e0-4c52-b69e-30ff505d9599 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:23.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1071/2149] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:07:23 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:07:23.790210 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308fb926-99d8-4299-a0ab-871669002a07 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:23.799872 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308fb926-99d8-4299-a0ab-871669002a07 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:23.804501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da003aab-89e2-4da1-a0f9-5f3ac1368327 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:23.805099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da003aab-89e2-4da1-a0f9-5f3ac1368327 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:23.805393 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da003aab-89e2-4da1-a0f9-5f3ac1368327 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:23.805666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da003aab-89e2-4da1-a0f9-5f3ac1368327 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:23.806301 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da003aab-89e2-4da1-a0f9-5f3ac1368327 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:23.807902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da003aab-89e2-4da1-a0f9-5f3ac1368327 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:23.818455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1079/2150] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:07:23 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:07:23.822442 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db634d3d-13e0-4c52-b69e-30ff505d9599 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:23.832095 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db634d3d-13e0-4c52-b69e-30ff505d9599 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:23.837520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fc4a4c-39e7-4016-b96c-f977723e4f59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:23.838157 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fc4a4c-39e7-4016-b96c-f977723e4f59 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:23.838434 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fc4a4c-39e7-4016-b96c-f977723e4f59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:23.838704 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fc4a4c-39e7-4016-b96c-f977723e4f59 admin admin] PATH_INFO: `/v3/projects/bcf6c738f8d842f99d2eec3b896c5b47/users/a6cf41aaddce4a86825bff00477793f9/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:23.854614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fc4a4c-39e7-4016-b96c-f977723e4f59 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bcf6c738f8d842f99d2eec3b896c5b47, user_id=a6cf41aaddce4a86825bff00477793f9, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:23.857252 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fc4a4c-39e7-4016-b96c-f977723e4f59 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:23.918536 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1072/2151] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:07:23 2022] PUT /identity/v3/projects/bcf6c738f8d842f99d2eec3b896c5b47/users/a6cf41aaddce4a86825bff00477793f9/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:07:23.923595 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f932fc-d97d-4066-a1a1-a90c7dd84f84 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:23.923880 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f932fc-d97d-4066-a1a1-a90c7dd84f84 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:23.924161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f932fc-d97d-4066-a1a1-a90c7dd84f84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:23.944800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-a4f932fc-d97d-4066-a1a1-a90c7dd84f84 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:23.946094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a4f932fc-d97d-4066-a1a1-a90c7dd84f84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:24.004144 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1080/2152] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:24.014417 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fc4a4c-39e7-4016-b96c-f977723e4f59 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:24.016101 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-87fc4a4c-39e7-4016-b96c-f977723e4f59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:24.047178 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:24.073184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fc4a4c-39e7-4016-b96c-f977723e4f59 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:24.076805 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704f32a1-4f6c-4707-a0a2-1ddc88753510 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:24.077429 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704f32a1-4f6c-4707-a0a2-1ddc88753510 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:24.077711 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704f32a1-4f6c-4707-a0a2-1ddc88753510 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:24.077978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704f32a1-4f6c-4707-a0a2-1ddc88753510 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:24.087736 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704f32a1-4f6c-4707-a0a2-1ddc88753510 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:24.089169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704f32a1-4f6c-4707-a0a2-1ddc88753510 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:24.098437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1073/2153] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:07:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:07:25.472571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e874899e-fc0a-4273-87d5-eb7cbab5a75f None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:25.472986 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e874899e-fc0a-4273-87d5-eb7cbab5a75f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:25.473135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e874899e-fc0a-4273-87d5-eb7cbab5a75f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:25.503836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-e874899e-fc0a-4273-87d5-eb7cbab5a75f None None] MFA Rules not processed for user `a6cf41aaddce4a86825bff00477793f9`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:25.505259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e874899e-fc0a-4273-87d5-eb7cbab5a75f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:25.594370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1081/2154] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:25 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:25.604255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704f32a1-4f6c-4707-a0a2-1ddc88753510 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:25.606002 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-704f32a1-4f6c-4707-a0a2-1ddc88753510 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:25.663235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704f32a1-4f6c-4707-a0a2-1ddc88753510 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:25.667055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfa0550-db77-4e07-8a89-9ed71e895a6e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:25.667667 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfa0550-db77-4e07-8a89-9ed71e895a6e service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:25.667926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfa0550-db77-4e07-8a89-9ed71e895a6e service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:25.668234 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfa0550-db77-4e07-8a89-9ed71e895a6e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:25.676698 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cfa0550-db77-4e07-8a89-9ed71e895a6e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:25.678285 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cfa0550-db77-4e07-8a89-9ed71e895a6e service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:25.689205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1074/2155] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:07:25 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:07:27.748106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e769c47-c1a6-49fe-bc10-648c03b8d71f None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:27.748587 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e769c47-c1a6-49fe-bc10-648c03b8d71f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:27.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e769c47-c1a6-49fe-bc10-648c03b8d71f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:27.756259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:27.768415 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9e769c47-c1a6-49fe-bc10-648c03b8d71f None None] MFA Rules not processed for user `a6cf41aaddce4a86825bff00477793f9`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:27.769726 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9e769c47-c1a6-49fe-bc10-648c03b8d71f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:27.777943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1082/2156] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:27 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:31.613007 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbe6471-5525-4886-83a8-e499770314c2 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:31.613919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbe6471-5525-4886-83a8-e499770314c2 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:31.613919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbe6471-5525-4886-83a8-e499770314c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:31.656334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-2bbe6471-5525-4886-83a8-e499770314c2 None None] MFA Rules not processed for user `bd13df3a2d0e4182bbd86dab7035c371`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:31.658265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2bbe6471-5525-4886-83a8-e499770314c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:31.719384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1075/2157] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:31 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Sep 04 01:07:38.025273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:40.966540 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9816f73-737f-4928-91ae-19bd7c3f3286 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:40.966540 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9816f73-737f-4928-91ae-19bd7c3f3286 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:40.966540 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9816f73-737f-4928-91ae-19bd7c3f3286 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:40.982358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-a9816f73-737f-4928-91ae-19bd7c3f3286 None None] MFA Rules not processed for user `f5e4723258c64ce5885154dded2d3d43`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:40.983584 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a9816f73-737f-4928-91ae-19bd7c3f3286 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:41.064455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1083/2158] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:07:40 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:41.712774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=65 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:51.049430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=25 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:07:54.620633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3855af05-909b-41ec-ad8f-9488f2a2923c None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:54.621242 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3855af05-909b-41ec-ad8f-9488f2a2923c None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:54.621242 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3855af05-909b-41ec-ad8f-9488f2a2923c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:54.647315 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-3855af05-909b-41ec-ad8f-9488f2a2923c None None] MFA Rules not processed for user `70cef7c271e046a3b49230057cc2d28f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:07:54.648828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-3855af05-909b-41ec-ad8f-9488f2a2923c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:54.716832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1076/2159] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 4 01:07:54 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:07:54.836334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9816f73-737f-4928-91ae-19bd7c3f3286 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:07:54.837867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a9816f73-737f-4928-91ae-19bd7c3f3286 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:07:54.925369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9816f73-737f-4928-91ae-19bd7c3f3286 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:07:54.931565 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be651172-91ab-4479-9434-2b16ed9fde04 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '123ab6ff796c4e599ce1d6f4095bd898', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:07:54.932256 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be651172-91ab-4479-9434-2b16ed9fde04 service swift] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:07:54.932704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be651172-91ab-4479-9434-2b16ed9fde04 service swift] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:07:54.932985 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be651172-91ab-4479-9434-2b16ed9fde04 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:07:54.942090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be651172-91ab-4479-9434-2b16ed9fde04 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:07:54.944288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be651172-91ab-4479-9434-2b16ed9fde04 service swift] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:07:54.955098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1084/2160] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 4 01:07:54 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 04 01:08:04.710568 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=25 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:08:04.950602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=32 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:08:06.189164 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5f4e96-1fee-48bb-a29a-07d5843fdb32 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:06.190155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5f4e96-1fee-48bb-a29a-07d5843fdb32 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:06.190467 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5f4e96-1fee-48bb-a29a-07d5843fdb32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:06.229600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-ce5f4e96-1fee-48bb-a29a-07d5843fdb32 None None] MFA Rules not processed for user `a6cf41aaddce4a86825bff00477793f9`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:06.231140 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ce5f4e96-1fee-48bb-a29a-07d5843fdb32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:06.243334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1077/2161] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:08:06 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:08:11.941873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be651172-91ab-4479-9434-2b16ed9fde04 service swift] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:11.943603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-be651172-91ab-4479-9434-2b16ed9fde04 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:11.949421 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5f4e96-1fee-48bb-a29a-07d5843fdb32 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:11.951323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ce5f4e96-1fee-48bb-a29a-07d5843fdb32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:12.074634 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5f4e96-1fee-48bb-a29a-07d5843fdb32 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:12.075112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be651172-91ab-4479-9434-2b16ed9fde04 service swift] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:12.080442 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921247c2-fa7b-42c3-8f5f-8ffd83994553 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b29f444d56ec49308f7fef16f4d1eea5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:12.081201 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921247c2-fa7b-42c3-8f5f-8ffd83994553 service placement] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:12.081514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84fe4ec4-02af-412e-b6cf-5a01d8efd7ff service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b29f444d56ec49308f7fef16f4d1eea5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:12.081697 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921247c2-fa7b-42c3-8f5f-8ffd83994553 service placement] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:12.082118 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921247c2-fa7b-42c3-8f5f-8ffd83994553 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:12.082279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fe4ec4-02af-412e-b6cf-5a01d8efd7ff service placement] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:12.082568 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fe4ec4-02af-412e-b6cf-5a01d8efd7ff service placement] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:12.082852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fe4ec4-02af-412e-b6cf-5a01d8efd7ff service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:12.084535 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-921247c2-fa7b-42c3-8f5f-8ffd83994553 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:12.084727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-84fe4ec4-02af-412e-b6cf-5a01d8efd7ff service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:12.099074 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921247c2-fa7b-42c3-8f5f-8ffd83994553 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:12.106761 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921247c2-fa7b-42c3-8f5f-8ffd83994553 service placement] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:12.107576 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84fe4ec4-02af-412e-b6cf-5a01d8efd7ff service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:12.108912 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84fe4ec4-02af-412e-b6cf-5a01d8efd7ff service placement] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:12.117118 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1078/2162] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:08:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:08:12.145291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1085/2163] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:08:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:08:13.538093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921247c2-fa7b-42c3-8f5f-8ffd83994553 service placement] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:13.539678 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-921247c2-fa7b-42c3-8f5f-8ffd83994553 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:13.556625 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921247c2-fa7b-42c3-8f5f-8ffd83994553 service placement] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:13.561846 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0274410c-0c19-49b7-a5e6-8bb9c98cb037 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:13.562627 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0274410c-0c19-49b7-a5e6-8bb9c98cb037 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:13.562913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0274410c-0c19-49b7-a5e6-8bb9c98cb037 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:13.563189 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0274410c-0c19-49b7-a5e6-8bb9c98cb037 admin admin] PATH_INFO: `/v3/users/d79e61978ff5400683c838d77a6ebb7e` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:13.564499 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0274410c-0c19-49b7-a5e6-8bb9c98cb037 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d79e61978ff5400683c838d77a6ebb7e)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:13.567874 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0274410c-0c19-49b7-a5e6-8bb9c98cb037 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:13.787207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0274410c-0c19-49b7-a5e6-8bb9c98cb037 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd79e61978ff5400683c838d77a6ebb7e'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:13.787871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0274410c-0c19-49b7-a5e6-8bb9c98cb037 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd79e61978ff5400683c838d77a6ebb7e'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:13.870747 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0274410c-0c19-49b7-a5e6-8bb9c98cb037 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd79e61978ff5400683c838d77a6ebb7e'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:13.884141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0274410c-0c19-49b7-a5e6-8bb9c98cb037 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd79e61978ff5400683c838d77a6ebb7e'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:13.884798 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0274410c-0c19-49b7-a5e6-8bb9c98cb037 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd79e61978ff5400683c838d77a6ebb7e'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:13.896861 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1079/2164] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:08:13 2022] DELETE /identity/v3/users/d79e61978ff5400683c838d77a6ebb7e => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:08:13.901426 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7010f046-8015-412e-8660-bc8655ca170b None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:13.906264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7010f046-8015-412e-8660-bc8655ca170b None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:13.906264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7010f046-8015-412e-8660-bc8655ca170b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:13.920470 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-7010f046-8015-412e-8660-bc8655ca170b None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:13.921688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7010f046-8015-412e-8660-bc8655ca170b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:13.984869 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1086/2165] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:08:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:08:13.995623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0274410c-0c19-49b7-a5e6-8bb9c98cb037 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:13.997021 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-0274410c-0c19-49b7-a5e6-8bb9c98cb037 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:14.065687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0274410c-0c19-49b7-a5e6-8bb9c98cb037 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:14.069362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5e3e9e-5bed-43a7-bb5f-4c6f194e7915 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:14.072347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5e3e9e-5bed-43a7-bb5f-4c6f194e7915 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:14.072347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5e3e9e-5bed-43a7-bb5f-4c6f194e7915 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:14.072347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5e3e9e-5bed-43a7-bb5f-4c6f194e7915 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:14.079936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5e3e9e-5bed-43a7-bb5f-4c6f194e7915 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:14.081576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5e3e9e-5bed-43a7-bb5f-4c6f194e7915 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:14.086096 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:08:14.092111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1080/2166] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:08:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:08:14.598976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7010f046-8015-412e-8660-bc8655ca170b None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:14.600499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7010f046-8015-412e-8660-bc8655ca170b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:14.637305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7010f046-8015-412e-8660-bc8655ca170b None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:14.644047 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6c162d-b040-41fa-ad1b-ea9e53e1303c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:14.645199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6c162d-b040-41fa-ad1b-ea9e53e1303c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:14.650381 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6c162d-b040-41fa-ad1b-ea9e53e1303c admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:14.650381 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6c162d-b040-41fa-ad1b-ea9e53e1303c admin admin] PATH_INFO: `/v3/projects/a0f43a31c7e549eeb636fcce144cb3bc` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:14.650381 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6c162d-b040-41fa-ad1b-ea9e53e1303c admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0f43a31c7e549eeb636fcce144cb3bc)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:14.651223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6c162d-b040-41fa-ad1b-ea9e53e1303c admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:14.826233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cc6c162d-b040-41fa-ad1b-ea9e53e1303c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0f43a31c7e549eeb636fcce144cb3bc', 'request_id': 'req-cc6c162d-b040-41fa-ad1b-ea9e53e1303c'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:14.833237 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cc6c162d-b040-41fa-ad1b-ea9e53e1303c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0f43a31c7e549eeb636fcce144cb3bc', 'request_id': 'req-cc6c162d-b040-41fa-ad1b-ea9e53e1303c'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:14.842679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cc6c162d-b040-41fa-ad1b-ea9e53e1303c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0f43a31c7e549eeb636fcce144cb3bc', 'request_id': 'req-cc6c162d-b040-41fa-ad1b-ea9e53e1303c'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:14.923868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cc6c162d-b040-41fa-ad1b-ea9e53e1303c admin admin] The token cache is being invalidate because project a0f43a31c7e549eeb636fcce144cb3bc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:08:14.924553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cc6c162d-b040-41fa-ad1b-ea9e53e1303c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:14.924981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cc6c162d-b040-41fa-ad1b-ea9e53e1303c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:14.928540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b6ada5-cdf2-4ef1-8117-30921bb2282f None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:14.929367 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b6ada5-cdf2-4ef1-8117-30921bb2282f None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:14.929640 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b6ada5-cdf2-4ef1-8117-30921bb2282f None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:14.931097 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1081/2168] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 4 01:08:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 01:08:14.931602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1087/2168] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:08:14 2022] DELETE /identity/v3/projects/a0f43a31c7e549eeb636fcce144cb3bc => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:08:14.932842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ea3bf2-b89c-42fb-b801-be61490bb204 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:14.933248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ea3bf2-b89c-42fb-b801-be61490bb204 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:14.933520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ea3bf2-b89c-42fb-b801-be61490bb204 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:14.938867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da629984-36e4-42f5-b16b-f9f0f513b396 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:14.939378 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da629984-36e4-42f5-b16b-f9f0f513b396 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:14.941464 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da629984-36e4-42f5-b16b-f9f0f513b396 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:14.971059 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-20ea3bf2-b89c-42fb-b801-be61490bb204 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:14.971779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-da629984-36e4-42f5-b16b-f9f0f513b396 None None] MFA Rules not processed for user `57c88f16c7df40adb917f49804fda21f`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:14.973351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-20ea3bf2-b89c-42fb-b801-be61490bb204 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:14.980041 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-da629984-36e4-42f5-b16b-f9f0f513b396 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:14.993689 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:08:15.075487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1082/2169] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:08:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:08:15.075487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70a3164-d167-422d-b325-ff421b1b970e None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:15.075487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70a3164-d167-422d-b325-ff421b1b970e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:15.075487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70a3164-d167-422d-b325-ff421b1b970e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:15.085163 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-a70a3164-d167-422d-b325-ff421b1b970e None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:15.085163 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a70a3164-d167-422d-b325-ff421b1b970e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:15.095935 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1083/2170] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:08:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:08:15.102072 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70a3164-d167-422d-b325-ff421b1b970e None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:15.119727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70a3164-d167-422d-b325-ff421b1b970e None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:15.127597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cdf5a2-dd36-4456-b711-8199facd2b80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:15.128205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cdf5a2-dd36-4456-b711-8199facd2b80 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:15.128485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cdf5a2-dd36-4456-b711-8199facd2b80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:15.129325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cdf5a2-dd36-4456-b711-8199facd2b80 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:15.130338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52cdf5a2-dd36-4456-b711-8199facd2b80 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:08:15.130463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52cdf5a2-dd36-4456-b711-8199facd2b80 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:15.132040 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52cdf5a2-dd36-4456-b711-8199facd2b80 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:15.139822 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1088/2171] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 4 01:08:14 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:08:15.150245 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1084/2172] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:08:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:08:15.154603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da629984-36e4-42f5-b16b-f9f0f513b396 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:15.157353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cdf5a2-dd36-4456-b711-8199facd2b80 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:15.159663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-52cdf5a2-dd36-4456-b711-8199facd2b80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:15.168443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da629984-36e4-42f5-b16b-f9f0f513b396 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:15.174098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d75b54-ebb1-4a45-81d2-537018e981fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:15.174720 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d75b54-ebb1-4a45-81d2-537018e981fb admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:15.175172 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d75b54-ebb1-4a45-81d2-537018e981fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:15.175479 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d75b54-ebb1-4a45-81d2-537018e981fb admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:15.176180 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d75b54-ebb1-4a45-81d2-537018e981fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:15.178873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d75b54-ebb1-4a45-81d2-537018e981fb admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:15.236975 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cdf5a2-dd36-4456-b711-8199facd2b80 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:15.241838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb44d34-6ea1-46ab-9f89-3171b59b8fe3 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '69153344fc4b45588bdbcecdfcfd7c3f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:15.242464 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb44d34-6ea1-46ab-9f89-3171b59b8fe3 service glance] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:15.242741 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb44d34-6ea1-46ab-9f89-3171b59b8fe3 service glance] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:15.243012 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb44d34-6ea1-46ab-9f89-3171b59b8fe3 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:15.254283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb44d34-6ea1-46ab-9f89-3171b59b8fe3 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:15.255837 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb44d34-6ea1-46ab-9f89-3171b59b8fe3 service glance] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:15.271705 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1085/2173] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:08:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:08:15.319362 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1089/2174] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:08:15 2022] POST /identity/v3/projects => generated 397 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:08:15.323556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb44d34-6ea1-46ab-9f89-3171b59b8fe3 service glance] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:15.334871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb44d34-6ea1-46ab-9f89-3171b59b8fe3 service glance] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:15.400798 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4eb4d42-3a9a-4489-8bf9-892d0770addf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:15.401514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4eb4d42-3a9a-4489-8bf9-892d0770addf admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:15.401796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4eb4d42-3a9a-4489-8bf9-892d0770addf admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:15.402070 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4eb4d42-3a9a-4489-8bf9-892d0770addf admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:15.402770 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4eb4d42-3a9a-4489-8bf9-892d0770addf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:15.404691 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4eb4d42-3a9a-4489-8bf9-892d0770addf admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:15.493515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1086/2175] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:08:15 2022] POST /identity/v3/users => generated 299 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:08:15.497600 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d75b54-ebb1-4a45-81d2-537018e981fb admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:15.508317 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d75b54-ebb1-4a45-81d2-537018e981fb admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:15.512946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f8ef57-c884-4eec-948f-5024795d3209 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:15.513562 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f8ef57-c884-4eec-948f-5024795d3209 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:15.513840 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f8ef57-c884-4eec-948f-5024795d3209 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:15.522575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f8ef57-c884-4eec-948f-5024795d3209 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:15.522575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f8ef57-c884-4eec-948f-5024795d3209 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:15.522575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f8ef57-c884-4eec-948f-5024795d3209 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:15.526871 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1090/2176] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:08:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:08:15.531198 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4eb4d42-3a9a-4489-8bf9-892d0770addf admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:15.545708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa8559a-dbfd-4efb-a038-88edb9809211 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:15.545708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa8559a-dbfd-4efb-a038-88edb9809211 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:15.545948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4eb4d42-3a9a-4489-8bf9-892d0770addf admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:15.546053 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa8559a-dbfd-4efb-a038-88edb9809211 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:15.546053 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1091/2177] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 4 01:08:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 01:08:15.547584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a7776a-c6ab-47b0-b634-2fa25874a9a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:15.548215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a7776a-c6ab-47b0-b634-2fa25874a9a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:15.548501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a7776a-c6ab-47b0-b634-2fa25874a9a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:15.548610 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f9a0f5-a100-45ce-8f16-e9161ad10e4f None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:15.548770 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a7776a-c6ab-47b0-b634-2fa25874a9a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:15.548876 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f9a0f5-a100-45ce-8f16-e9161ad10e4f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:15.549188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f9a0f5-a100-45ce-8f16-e9161ad10e4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:15.549563 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a7776a-c6ab-47b0-b634-2fa25874a9a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:15.551547 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a7776a-c6ab-47b0-b634-2fa25874a9a6 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:15.557584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:08:15.562889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1087/2178] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:08:15 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:08:15.566870 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a7776a-c6ab-47b0-b634-2fa25874a9a6 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:15.574059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-84f9a0f5-a100-45ce-8f16-e9161ad10e4f None None] MFA Rules not processed for user `da4f44b86ca94adf959d9a41d915c902`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:15.575280 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-84f9a0f5-a100-45ce-8f16-e9161ad10e4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:15.578688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a7776a-c6ab-47b0-b634-2fa25874a9a6 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:15.584330 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5dad20-4dc8-4f96-a478-1624708e797c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:15.585026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5dad20-4dc8-4f96-a478-1624708e797c admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:15.585291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5dad20-4dc8-4f96-a478-1624708e797c admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:15.585899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5dad20-4dc8-4f96-a478-1624708e797c admin admin] PATH_INFO: `/v3/projects/f24e82fca13f44f390d817aa4453e2d3/users/2eeacb166acb4bfea6aa820a7fcdb841/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:15.601531 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5dad20-4dc8-4f96-a478-1624708e797c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f24e82fca13f44f390d817aa4453e2d3, user_id=2eeacb166acb4bfea6aa820a7fcdb841, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:15.604197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5dad20-4dc8-4f96-a478-1624708e797c admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:15.629377 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1092/2179] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 4 01:08:15 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:08:15.642081 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f9a0f5-a100-45ce-8f16-e9161ad10e4f None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:15.643424 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-84f9a0f5-a100-45ce-8f16-e9161ad10e4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:15.654626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f9a0f5-a100-45ce-8f16-e9161ad10e4f None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:15.660316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f6cdb5-c631-4ef9-a37e-8fd09176ea3c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:15.660316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f6cdb5-c631-4ef9-a37e-8fd09176ea3c service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:15.660316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f6cdb5-c631-4ef9-a37e-8fd09176ea3c service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:15.660316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f6cdb5-c631-4ef9-a37e-8fd09176ea3c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:15.666420 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f6cdb5-c631-4ef9-a37e-8fd09176ea3c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:15.667867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f6cdb5-c631-4ef9-a37e-8fd09176ea3c service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:15.675808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1093/2180] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:08:15 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:08:15.682660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1088/2181] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:08:15 2022] PUT /identity/v3/projects/f24e82fca13f44f390d817aa4453e2d3/users/2eeacb166acb4bfea6aa820a7fcdb841/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:08:15.700498 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896159f2-1bf3-47d7-b3ea-045ed43352bc None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:15.700784 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896159f2-1bf3-47d7-b3ea-045ed43352bc None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:15.701053 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896159f2-1bf3-47d7-b3ea-045ed43352bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:15.731691 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-896159f2-1bf3-47d7-b3ea-045ed43352bc None None] MFA Rules not processed for user `2eeacb166acb4bfea6aa820a7fcdb841`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:15.732899 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-896159f2-1bf3-47d7-b3ea-045ed43352bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:15.760456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260d2d5c-7442-4c20-9b97-a5b8b7a4b519 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:15.760744 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260d2d5c-7442-4c20-9b97-a5b8b7a4b519 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:15.761015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260d2d5c-7442-4c20-9b97-a5b8b7a4b519 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:15.762440 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1089/2182] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 4 01:08:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 04 01:08:15.767832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145deecd-20a3-4d51-bf3e-02a3ed8bb3e1 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:15.769207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145deecd-20a3-4d51-bf3e-02a3ed8bb3e1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:15.770247 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145deecd-20a3-4d51-bf3e-02a3ed8bb3e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:15.789966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-145deecd-20a3-4d51-bf3e-02a3ed8bb3e1 None None] MFA Rules not processed for user `57c88f16c7df40adb917f49804fda21f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:15.791260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-145deecd-20a3-4d51-bf3e-02a3ed8bb3e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:15.795287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:08:15.800379 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1094/2183] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:08:15 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:08:15.812120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896159f2-1bf3-47d7-b3ea-045ed43352bc None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:15.813609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-896159f2-1bf3-47d7-b3ea-045ed43352bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:15.868949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1090/2184] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 4 01:08:15 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:08:15.873200 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896159f2-1bf3-47d7-b3ea-045ed43352bc None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:15.876863 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ab9f59-732b-44c7-bf08-0a926e409fbb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:15.877495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ab9f59-732b-44c7-bf08-0a926e409fbb service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:15.877781 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ab9f59-732b-44c7-bf08-0a926e409fbb service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:15.878054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ab9f59-732b-44c7-bf08-0a926e409fbb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:15.880765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145deecd-20a3-4d51-bf3e-02a3ed8bb3e1 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:15.887833 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ab9f59-732b-44c7-bf08-0a926e409fbb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:15.889324 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ab9f59-732b-44c7-bf08-0a926e409fbb service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:15.899474 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1095/2185] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:08:15 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:08:15.936744 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145deecd-20a3-4d51-bf3e-02a3ed8bb3e1 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:15.940402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e44aa4-9a1b-4409-a753-239bc225bb83 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:15.942003 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e44aa4-9a1b-4409-a753-239bc225bb83 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:15.942290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e44aa4-9a1b-4409-a753-239bc225bb83 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:15.942548 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e44aa4-9a1b-4409-a753-239bc225bb83 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:15.952620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e44aa4-9a1b-4409-a753-239bc225bb83 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:15.954499 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e44aa4-9a1b-4409-a753-239bc225bb83 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:15.966076 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1091/2186] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:08:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:08:25.072938 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ab9f59-732b-44c7-bf08-0a926e409fbb service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:25.140785 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ab9f59-732b-44c7-bf08-0a926e409fbb service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:25.146829 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf71fb55-31bc-4f47-9457-38d253527f27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:25.147463 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf71fb55-31bc-4f47-9457-38d253527f27 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:25.147744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf71fb55-31bc-4f47-9457-38d253527f27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:25.148015 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf71fb55-31bc-4f47-9457-38d253527f27 admin admin] PATH_INFO: `/v3/users/2eeacb166acb4bfea6aa820a7fcdb841` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:25.149195 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf71fb55-31bc-4f47-9457-38d253527f27 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2eeacb166acb4bfea6aa820a7fcdb841)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:25.151131 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf71fb55-31bc-4f47-9457-38d253527f27 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:25.332801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bf71fb55-31bc-4f47-9457-38d253527f27 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2eeacb166acb4bfea6aa820a7fcdb841'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:25.396364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bf71fb55-31bc-4f47-9457-38d253527f27 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2eeacb166acb4bfea6aa820a7fcdb841'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:25.408655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bf71fb55-31bc-4f47-9457-38d253527f27 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2eeacb166acb4bfea6aa820a7fcdb841'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:25.409369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bf71fb55-31bc-4f47-9457-38d253527f27 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2eeacb166acb4bfea6aa820a7fcdb841'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:25.416323 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bf71fb55-31bc-4f47-9457-38d253527f27 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2eeacb166acb4bfea6aa820a7fcdb841'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:25.422447 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1096/2187] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:08:25 2022] DELETE /identity/v3/users/2eeacb166acb4bfea6aa820a7fcdb841 => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:08:25.427030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98664c3f-caf5-45cb-b5a0-d76d2536aa01 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:25.427520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98664c3f-caf5-45cb-b5a0-d76d2536aa01 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:25.427615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98664c3f-caf5-45cb-b5a0-d76d2536aa01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:25.446666 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-98664c3f-caf5-45cb-b5a0-d76d2536aa01 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:25.447864 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-98664c3f-caf5-45cb-b5a0-d76d2536aa01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:25.506346 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1092/2188] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:08:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:08:25.515999 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf71fb55-31bc-4f47-9457-38d253527f27 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:25.517446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-bf71fb55-31bc-4f47-9457-38d253527f27 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:25.539559 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:08:25.574538 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf71fb55-31bc-4f47-9457-38d253527f27 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:25.578222 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae50072-0e2f-413c-b638-72ab8e04b077 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:25.578823 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae50072-0e2f-413c-b638-72ab8e04b077 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:25.579119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae50072-0e2f-413c-b638-72ab8e04b077 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:25.579374 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae50072-0e2f-413c-b638-72ab8e04b077 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:25.589626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae50072-0e2f-413c-b638-72ab8e04b077 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:25.591054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae50072-0e2f-413c-b638-72ab8e04b077 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:25.600385 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1097/2189] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:08:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:08:26.032148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98664c3f-caf5-45cb-b5a0-d76d2536aa01 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:26.033607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-98664c3f-caf5-45cb-b5a0-d76d2536aa01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:26.046227 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98664c3f-caf5-45cb-b5a0-d76d2536aa01 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:26.051199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec43b74-0e5c-4179-b46b-95a3188ffe28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:26.051838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec43b74-0e5c-4179-b46b-95a3188ffe28 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:26.052119 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec43b74-0e5c-4179-b46b-95a3188ffe28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:26.052391 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec43b74-0e5c-4179-b46b-95a3188ffe28 admin admin] PATH_INFO: `/v3/projects/f24e82fca13f44f390d817aa4453e2d3` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:26.053552 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec43b74-0e5c-4179-b46b-95a3188ffe28 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f24e82fca13f44f390d817aa4453e2d3)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:26.055460 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec43b74-0e5c-4179-b46b-95a3188ffe28 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:26.062754 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:08:26.153952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-cec43b74-0e5c-4179-b46b-95a3188ffe28 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f24e82fca13f44f390d817aa4453e2d3', 'request_id': 'req-cec43b74-0e5c-4179-b46b-95a3188ffe28'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:26.159661 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-cec43b74-0e5c-4179-b46b-95a3188ffe28 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f24e82fca13f44f390d817aa4453e2d3', 'request_id': 'req-cec43b74-0e5c-4179-b46b-95a3188ffe28'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:26.208595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-cec43b74-0e5c-4179-b46b-95a3188ffe28 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f24e82fca13f44f390d817aa4453e2d3', 'request_id': 'req-cec43b74-0e5c-4179-b46b-95a3188ffe28'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:26.220253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-cec43b74-0e5c-4179-b46b-95a3188ffe28 admin admin] The token cache is being invalidate because project f24e82fca13f44f390d817aa4453e2d3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:08:26.220732 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-cec43b74-0e5c-4179-b46b-95a3188ffe28 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:26.221255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-cec43b74-0e5c-4179-b46b-95a3188ffe28 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:08:26.223354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1093/2190] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:08:26 2022] DELETE /identity/v3/projects/f24e82fca13f44f390d817aa4453e2d3 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:08:26.228610 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a2e597-9502-40b2-a5b9-6840d50608b8 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:26.229126 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a2e597-9502-40b2-a5b9-6840d50608b8 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:26.229244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a2e597-9502-40b2-a5b9-6840d50608b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:26.249561 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-54a2e597-9502-40b2-a5b9-6840d50608b8 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:26.251090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-54a2e597-9502-40b2-a5b9-6840d50608b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:26.311315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1098/2191] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:08:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:08:26.319193 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e672b4c5-ec22-45f3-adfd-4fb60a47ffd0 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:26.319476 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e672b4c5-ec22-45f3-adfd-4fb60a47ffd0 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:26.319750 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e672b4c5-ec22-45f3-adfd-4fb60a47ffd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:26.340167 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-e672b4c5-ec22-45f3-adfd-4fb60a47ffd0 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:26.341408 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e672b4c5-ec22-45f3-adfd-4fb60a47ffd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:26.349958 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1094/2192] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:08:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:08:26.354297 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a2e597-9502-40b2-a5b9-6840d50608b8 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:26.365219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a2e597-9502-40b2-a5b9-6840d50608b8 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:26.369755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5b6cfb-747e-49fa-9fb7-9058d6f2c680 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:26.370345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b6cfb-747e-49fa-9fb7-9058d6f2c680 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:26.370626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b6cfb-747e-49fa-9fb7-9058d6f2c680 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:26.370901 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5b6cfb-747e-49fa-9fb7-9058d6f2c680 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:26.371553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5b6cfb-747e-49fa-9fb7-9058d6f2c680 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:08:26.371857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5b6cfb-747e-49fa-9fb7-9058d6f2c680 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:26.373490 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5b6cfb-747e-49fa-9fb7-9058d6f2c680 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:26.385201 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1099/2193] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:08:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:08:26.389321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e672b4c5-ec22-45f3-adfd-4fb60a47ffd0 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:26.398622 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e672b4c5-ec22-45f3-adfd-4fb60a47ffd0 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:26.403781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c409ff-0882-4eb0-a3c8-70c90eff64ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:26.404397 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c409ff-0882-4eb0-a3c8-70c90eff64ee admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:26.404632 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c409ff-0882-4eb0-a3c8-70c90eff64ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:26.404900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c409ff-0882-4eb0-a3c8-70c90eff64ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:26.405554 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c409ff-0882-4eb0-a3c8-70c90eff64ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:26.407410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c409ff-0882-4eb0-a3c8-70c90eff64ee admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:26.515664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1095/2194] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:08:26 2022] POST /identity/v3/projects => generated 371 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:08:26.516896 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5b6cfb-747e-49fa-9fb7-9058d6f2c680 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:26.527736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5b6cfb-747e-49fa-9fb7-9058d6f2c680 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:26.588709 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02dd436b-3f95-473a-b0b1-b5a04a5fe1a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:26.589393 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dd436b-3f95-473a-b0b1-b5a04a5fe1a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:26.589674 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dd436b-3f95-473a-b0b1-b5a04a5fe1a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:26.589947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dd436b-3f95-473a-b0b1-b5a04a5fe1a6 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:26.590586 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02dd436b-3f95-473a-b0b1-b5a04a5fe1a6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:26.592573 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02dd436b-3f95-473a-b0b1-b5a04a5fe1a6 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:26.698959 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1100/2195] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:08:26 2022] POST /identity/v3/users => generated 286 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:08:26.703167 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c409ff-0882-4eb0-a3c8-70c90eff64ee admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:26.714177 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c409ff-0882-4eb0-a3c8-70c90eff64ee admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:26.718846 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443a67fc-2b09-46ed-9983-8e4fa0045819 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:26.719438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443a67fc-2b09-46ed-9983-8e4fa0045819 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:26.719716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443a67fc-2b09-46ed-9983-8e4fa0045819 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:26.719989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443a67fc-2b09-46ed-9983-8e4fa0045819 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:26.720631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443a67fc-2b09-46ed-9983-8e4fa0045819 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:26.722957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443a67fc-2b09-46ed-9983-8e4fa0045819 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:26.733041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1096/2196] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:08:26 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:08:26.737253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02dd436b-3f95-473a-b0b1-b5a04a5fe1a6 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:26.747296 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02dd436b-3f95-473a-b0b1-b5a04a5fe1a6 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:26.752007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff3a1b6-99ec-42d5-ac31-b6f61362159d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:26.752593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff3a1b6-99ec-42d5-ac31-b6f61362159d admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:26.752934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff3a1b6-99ec-42d5-ac31-b6f61362159d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:26.753232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff3a1b6-99ec-42d5-ac31-b6f61362159d admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:26.753868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff3a1b6-99ec-42d5-ac31-b6f61362159d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:26.755619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff3a1b6-99ec-42d5-ac31-b6f61362159d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:26.766296 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1101/2197] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:08:26 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:08:26.773672 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443a67fc-2b09-46ed-9983-8e4fa0045819 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:26.785983 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443a67fc-2b09-46ed-9983-8e4fa0045819 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:26.792571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e592af7e-d357-4645-8e7b-b2b59d367768 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:26.793544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e592af7e-d357-4645-8e7b-b2b59d367768 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:26.793966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e592af7e-d357-4645-8e7b-b2b59d367768 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:26.794383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e592af7e-d357-4645-8e7b-b2b59d367768 admin admin] PATH_INFO: `/v3/projects/735896cd6b3b4c448dd94b3bf7577cb8/users/13182cef4cf8458ab34779ffd7514c6c/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:26.812424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e592af7e-d357-4645-8e7b-b2b59d367768 admin admin] RBAC: Authorizing `identity:create_grant(project_id=735896cd6b3b4c448dd94b3bf7577cb8, user_id=13182cef4cf8458ab34779ffd7514c6c, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:26.815951 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e592af7e-d357-4645-8e7b-b2b59d367768 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:26.887129 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1097/2198] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:08:26 2022] PUT /identity/v3/projects/735896cd6b3b4c448dd94b3bf7577cb8/users/13182cef4cf8458ab34779ffd7514c6c/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:08:26.892068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a6f6ea-dc68-4fb6-8e57-2ecefbdf5a1a None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:26.892379 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a6f6ea-dc68-4fb6-8e57-2ecefbdf5a1a None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:26.892650 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a6f6ea-dc68-4fb6-8e57-2ecefbdf5a1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:26.912176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-c4a6f6ea-dc68-4fb6-8e57-2ecefbdf5a1a None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:26.913436 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c4a6f6ea-dc68-4fb6-8e57-2ecefbdf5a1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:26.942013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:08:26.990950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1102/2199] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:08:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:08:26.997327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e592af7e-d357-4645-8e7b-b2b59d367768 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:26.999643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e592af7e-d357-4645-8e7b-b2b59d367768 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:27.070749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e592af7e-d357-4645-8e7b-b2b59d367768 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:27.075060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c67375-28ec-4281-a257-af322e9c8fe4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:27.075819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c67375-28ec-4281-a257-af322e9c8fe4 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:27.076351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c67375-28ec-4281-a257-af322e9c8fe4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:27.076691 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c67375-28ec-4281-a257-af322e9c8fe4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:27.088554 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c67375-28ec-4281-a257-af322e9c8fe4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:27.090284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c67375-28ec-4281-a257-af322e9c8fe4 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:27.100764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1098/2200] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:08:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:08:28.156943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9284cc19-364b-4c2e-8d67-4e7d0523de91 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:28.157485 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9284cc19-364b-4c2e-8d67-4e7d0523de91 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:28.157566 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9284cc19-364b-4c2e-8d67-4e7d0523de91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:28.177902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9284cc19-364b-4c2e-8d67-4e7d0523de91 None None] MFA Rules not processed for user `70cef7c271e046a3b49230057cc2d28f`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:28.179105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9284cc19-364b-4c2e-8d67-4e7d0523de91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:28.241913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1103/2201] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 4 01:08:28 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:08:28.257459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c67375-28ec-4281-a257-af322e9c8fe4 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:28.259723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a3c67375-28ec-4281-a257-af322e9c8fe4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:28.300274 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:08:28.344983 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c67375-28ec-4281-a257-af322e9c8fe4 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:28.352231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb2d1d6-661e-4a66-a52a-9f939fdc571f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '123ab6ff796c4e599ce1d6f4095bd898', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:28.352904 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb2d1d6-661e-4a66-a52a-9f939fdc571f service swift] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:28.353252 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb2d1d6-661e-4a66-a52a-9f939fdc571f service swift] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:28.353550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb2d1d6-661e-4a66-a52a-9f939fdc571f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:28.368489 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb2d1d6-661e-4a66-a52a-9f939fdc571f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:28.370124 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb2d1d6-661e-4a66-a52a-9f939fdc571f service swift] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:28.380395 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1099/2202] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 4 01:08:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 04 01:08:35.199455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6283e768-c437-4f6d-a0c5-48f83d7768b4 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:35.199455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6283e768-c437-4f6d-a0c5-48f83d7768b4 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:35.200038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6283e768-c437-4f6d-a0c5-48f83d7768b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:35.233607 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-6283e768-c437-4f6d-a0c5-48f83d7768b4 None None] MFA Rules not processed for user `13182cef4cf8458ab34779ffd7514c6c`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:35.236151 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-6283e768-c437-4f6d-a0c5-48f83d7768b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:35.337747 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1104/2203] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:08:35 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:08:35.342119 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb2d1d6-661e-4a66-a52a-9f939fdc571f service swift] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:35.358607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb2d1d6-661e-4a66-a52a-9f939fdc571f service swift] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:35.369362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef8997e-61cb-430d-b15e-33c0f86f913b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:35.370088 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef8997e-61cb-430d-b15e-33c0f86f913b admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:35.370444 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef8997e-61cb-430d-b15e-33c0f86f913b admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:35.370792 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef8997e-61cb-430d-b15e-33c0f86f913b admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:35.371615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef8997e-61cb-430d-b15e-33c0f86f913b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:35.374045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef8997e-61cb-430d-b15e-33c0f86f913b admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:35.478402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f11118-72ae-423a-9efc-f982fe31158e None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:35.478718 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f11118-72ae-423a-9efc-f982fe31158e None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:35.479026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f11118-72ae-423a-9efc-f982fe31158e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:35.518689 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-a3f11118-72ae-423a-9efc-f982fe31158e None None] MFA Rules not processed for user `77de7bb68f3c45d680c8a78dc40df07b`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:35.526112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a3f11118-72ae-423a-9efc-f982fe31158e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:35.552918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1100/2204] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:08:35 2022] POST /identity/v3/projects => generated 373 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:08:35.555661 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef8997e-61cb-430d-b15e-33c0f86f913b admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:35.614861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1105/2205] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:08:35 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:08:35.713546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef8997e-61cb-430d-b15e-33c0f86f913b admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:35.727133 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e92a98-ecf5-4ca6-bcd8-3b4dfce1452d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:35.727837 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e92a98-ecf5-4ca6-bcd8-3b4dfce1452d admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:35.729388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e92a98-ecf5-4ca6-bcd8-3b4dfce1452d admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:35.729388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e92a98-ecf5-4ca6-bcd8-3b4dfce1452d admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:35.729388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e92a98-ecf5-4ca6-bcd8-3b4dfce1452d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:35.730980 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e92a98-ecf5-4ca6-bcd8-3b4dfce1452d admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:35.888169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1101/2206] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:08:35 2022] POST /identity/v3/users => generated 293 bytes in 334 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:08:35.892041 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f11118-72ae-423a-9efc-f982fe31158e None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:35.905911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f11118-72ae-423a-9efc-f982fe31158e None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:35.915949 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf85793-d9f8-4d63-9d3b-96b5ef7b99b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:35.926783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf85793-d9f8-4d63-9d3b-96b5ef7b99b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:35.929439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf85793-d9f8-4d63-9d3b-96b5ef7b99b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:35.929894 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf85793-d9f8-4d63-9d3b-96b5ef7b99b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:35.932079 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf85793-d9f8-4d63-9d3b-96b5ef7b99b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:35.935731 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf85793-d9f8-4d63-9d3b-96b5ef7b99b3 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:35.943056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:08:35.962418 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1106/2207] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:08:35 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:08:35.963407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e92a98-ecf5-4ca6-bcd8-3b4dfce1452d admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:35.975682 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e92a98-ecf5-4ca6-bcd8-3b4dfce1452d admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:35.980511 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6111e458-117e-4b21-8353-0482b25ce3d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:35.981229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6111e458-117e-4b21-8353-0482b25ce3d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:35.981517 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6111e458-117e-4b21-8353-0482b25ce3d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:35.981932 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6111e458-117e-4b21-8353-0482b25ce3d6 admin admin] PATH_INFO: `/v3/projects/2e0ecc6513e54f9597cef651eabc4879/users/4306e3c30d854718a031bee8020cc83d/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:36.016633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6111e458-117e-4b21-8353-0482b25ce3d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e0ecc6513e54f9597cef651eabc4879, user_id=4306e3c30d854718a031bee8020cc83d, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:36.020372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6111e458-117e-4b21-8353-0482b25ce3d6 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:36.107220 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1102/2208] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:08:35 2022] PUT /identity/v3/projects/2e0ecc6513e54f9597cef651eabc4879/users/4306e3c30d854718a031bee8020cc83d/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:08:36.110797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf85793-d9f8-4d63-9d3b-96b5ef7b99b3 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:36.129861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf85793-d9f8-4d63-9d3b-96b5ef7b99b3 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:36.259470 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb08f255-6988-45c7-b6f4-56549c5cdd28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:36.260159 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb08f255-6988-45c7-b6f4-56549c5cdd28 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:36.260299 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb08f255-6988-45c7-b6f4-56549c5cdd28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:36.261017 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb08f255-6988-45c7-b6f4-56549c5cdd28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:36.261247 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb08f255-6988-45c7-b6f4-56549c5cdd28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:36.262868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb08f255-6988-45c7-b6f4-56549c5cdd28 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:36.290429 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1107/2209] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:08:36 2022] GET /identity/v3/roles => generated 1420 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:08:36.291036 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6111e458-117e-4b21-8353-0482b25ce3d6 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:36.306511 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6111e458-117e-4b21-8353-0482b25ce3d6 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:36.319899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c2fe08-68a6-4bf9-9f44-73718db587a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:36.320617 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c2fe08-68a6-4bf9-9f44-73718db587a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:36.320900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c2fe08-68a6-4bf9-9f44-73718db587a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:36.321200 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c2fe08-68a6-4bf9-9f44-73718db587a1 admin admin] PATH_INFO: `/v3/projects/2e0ecc6513e54f9597cef651eabc4879/users/4306e3c30d854718a031bee8020cc83d/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:36.323942 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c2fe08-68a6-4bf9-9f44-73718db587a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e0ecc6513e54f9597cef651eabc4879, user_id=4306e3c30d854718a031bee8020cc83d, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:36.326622 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c2fe08-68a6-4bf9-9f44-73718db587a1 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:36.363455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1103/2210] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:08:36 2022] PUT /identity/v3/projects/2e0ecc6513e54f9597cef651eabc4879/users/4306e3c30d854718a031bee8020cc83d/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:08:44.297394 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd15763-3c3f-4d7b-86c9-43c64da4ede6 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:44.298032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd15763-3c3f-4d7b-86c9-43c64da4ede6 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:44.298032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd15763-3c3f-4d7b-86c9-43c64da4ede6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:44.327846 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9dd15763-3c3f-4d7b-86c9-43c64da4ede6 None None] MFA Rules not processed for user `4306e3c30d854718a031bee8020cc83d`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:44.329095 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9dd15763-3c3f-4d7b-86c9-43c64da4ede6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:44.410400 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1108/2211] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:08:44 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:08:44.423891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c2fe08-68a6-4bf9-9f44-73718db587a1 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:44.428599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e1c2fe08-68a6-4bf9-9f44-73718db587a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:44.480103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:08:44.501165 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c2fe08-68a6-4bf9-9f44-73718db587a1 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:44.506645 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8604ba3f-cd46-4919-aa27-88313b354662 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:44.507302 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8604ba3f-cd46-4919-aa27-88313b354662 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:44.507583 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8604ba3f-cd46-4919-aa27-88313b354662 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:44.507860 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8604ba3f-cd46-4919-aa27-88313b354662 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:44.568777 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8604ba3f-cd46-4919-aa27-88313b354662 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:44.570041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8604ba3f-cd46-4919-aa27-88313b354662 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:44.579581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1104/2212] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:08:44 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:08:47.329545 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a7badc-823f-410f-825c-1b6285fae8ba None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:47.330118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a7badc-823f-410f-825c-1b6285fae8ba None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:47.330118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a7badc-823f-410f-825c-1b6285fae8ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:47.356481 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-c4a7badc-823f-410f-825c-1b6285fae8ba None None] MFA Rules not processed for user `13182cef4cf8458ab34779ffd7514c6c`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:47.358159 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c4a7badc-823f-410f-825c-1b6285fae8ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:47.368484 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1109/2213] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:08:47 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:08:54.575894 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=32 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:08:57.348849 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=67 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:08:57.414886 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba8c8c9-631e-4781-a5da-5da92a8617ed None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:57.415382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba8c8c9-631e-4781-a5da-5da92a8617ed None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:57.415496 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba8c8c9-631e-4781-a5da-5da92a8617ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:57.448062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-fba8c8c9-631e-4781-a5da-5da92a8617ed None None] MFA Rules not processed for user `70cef7c271e046a3b49230057cc2d28f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:57.449447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-fba8c8c9-631e-4781-a5da-5da92a8617ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:57.529730 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1105/2214] 10.222.0.42 () {58 vars in 967 bytes} [Sun Sep 4 01:08:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:08:57.565150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a7badc-823f-410f-825c-1b6285fae8ba None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:57.642997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a7badc-823f-410f-825c-1b6285fae8ba None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:57.642997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b0600c-d2ca-4c73-b965-e7181f64bb2c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '123ab6ff796c4e599ce1d6f4095bd898', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:57.642997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b0600c-d2ca-4c73-b965-e7181f64bb2c service swift] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:57.642997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b0600c-d2ca-4c73-b965-e7181f64bb2c service swift] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:57.643509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b0600c-d2ca-4c73-b965-e7181f64bb2c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:57.651863 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b0600c-d2ca-4c73-b965-e7181f64bb2c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:57.653515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b0600c-d2ca-4c73-b965-e7181f64bb2c service swift] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:57.663233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1110/2215] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 4 01:08:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 04 01:08:58.698989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cef51d-652a-47ed-84eb-a5db1480c3be None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:58.699483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cef51d-652a-47ed-84eb-a5db1480c3be None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:58.699626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cef51d-652a-47ed-84eb-a5db1480c3be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:58.725602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-19cef51d-652a-47ed-84eb-a5db1480c3be None None] MFA Rules not processed for user `70cef7c271e046a3b49230057cc2d28f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:08:58.726736 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-19cef51d-652a-47ed-84eb-a5db1480c3be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:08:58.736163 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1106/2216] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 4 01:08:58 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:08:58.752010 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b0600c-d2ca-4c73-b965-e7181f64bb2c service swift] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:08:58.763452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b0600c-d2ca-4c73-b965-e7181f64bb2c service swift] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:08:58.768525 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3a6d77-34cc-4355-b96e-e1bef57ae715 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '123ab6ff796c4e599ce1d6f4095bd898', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:08:58.769145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3a6d77-34cc-4355-b96e-e1bef57ae715 service swift] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:08:58.769460 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3a6d77-34cc-4355-b96e-e1bef57ae715 service swift] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:08:58.769739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3a6d77-34cc-4355-b96e-e1bef57ae715 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:08:58.778339 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3a6d77-34cc-4355-b96e-e1bef57ae715 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:08:58.779965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3a6d77-34cc-4355-b96e-e1bef57ae715 service swift] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:08:58.788252 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1111/2217] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 4 01:08:58 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 04 01:09:08.718029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=28 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:09:08.784893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=38 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:09:20.758800 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103c3aad-4506-45f8-b5f6-a1181032a714 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:09:20.758800 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103c3aad-4506-45f8-b5f6-a1181032a714 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:09:20.758800 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103c3aad-4506-45f8-b5f6-a1181032a714 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:09:20.782522 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-103c3aad-4506-45f8-b5f6-a1181032a714 None None] MFA Rules not processed for user `70cef7c271e046a3b49230057cc2d28f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:09:20.784904 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-103c3aad-4506-45f8-b5f6-a1181032a714 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:09:20.794395 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1107/2218] 10.222.0.42 () {58 vars in 967 bytes} [Sun Sep 4 01:09:20 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:09:20.810458 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3a6d77-34cc-4355-b96e-e1bef57ae715 service swift] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:09:20.855701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3a6d77-34cc-4355-b96e-e1bef57ae715 service swift] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:09:20.866164 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8716ecd4-5b3f-4d12-84ca-553ff994b36d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '123ab6ff796c4e599ce1d6f4095bd898', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:09:20.868098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8716ecd4-5b3f-4d12-84ca-553ff994b36d service swift] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:09:20.868098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8716ecd4-5b3f-4d12-84ca-553ff994b36d service swift] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:09:20.868098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8716ecd4-5b3f-4d12-84ca-553ff994b36d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:09:20.890387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8716ecd4-5b3f-4d12-84ca-553ff994b36d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:09:20.892313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8716ecd4-5b3f-4d12-84ca-553ff994b36d service swift] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:09:20.911208 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1112/2219] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 4 01:09:20 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 04 01:09:24.430424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103c3aad-4506-45f8-b5f6-a1181032a714 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:09:24.432022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-103c3aad-4506-45f8-b5f6-a1181032a714 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:09:24.518621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103c3aad-4506-45f8-b5f6-a1181032a714 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:09:24.523823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314c74a0-fbce-472c-85a4-d582e4908a04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:09:24.524488 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314c74a0-fbce-472c-85a4-d582e4908a04 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:09:24.524762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314c74a0-fbce-472c-85a4-d582e4908a04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:09:24.525029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314c74a0-fbce-472c-85a4-d582e4908a04 admin admin] PATH_INFO: `/v3/users/a6cf41aaddce4a86825bff00477793f9` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:09:24.526240 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314c74a0-fbce-472c-85a4-d582e4908a04 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a6cf41aaddce4a86825bff00477793f9)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:09:24.528321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314c74a0-fbce-472c-85a4-d582e4908a04 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:09:24.710099 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-314c74a0-fbce-472c-85a4-d582e4908a04 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a6cf41aaddce4a86825bff00477793f9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:09:24.711397 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-314c74a0-fbce-472c-85a4-d582e4908a04 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a6cf41aaddce4a86825bff00477793f9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:09:24.791877 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-314c74a0-fbce-472c-85a4-d582e4908a04 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a6cf41aaddce4a86825bff00477793f9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:09:24.807079 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-314c74a0-fbce-472c-85a4-d582e4908a04 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a6cf41aaddce4a86825bff00477793f9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:09:24.807843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-314c74a0-fbce-472c-85a4-d582e4908a04 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a6cf41aaddce4a86825bff00477793f9'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:09:24.822456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1108/2220] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:09:24 2022] DELETE /identity/v3/users/a6cf41aaddce4a86825bff00477793f9 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:09:25.101830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8716ecd4-5b3f-4d12-84ca-553ff994b36d service swift] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:09:25.113724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8716ecd4-5b3f-4d12-84ca-553ff994b36d service swift] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:09:25.202639 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04dae9f-bb1e-426d-82e3-e490f31a549f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:09:25.203269 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04dae9f-bb1e-426d-82e3-e490f31a549f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:09:25.203547 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04dae9f-bb1e-426d-82e3-e490f31a549f admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:09:25.203819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04dae9f-bb1e-426d-82e3-e490f31a549f admin admin] PATH_INFO: `/v3/projects/bcf6c738f8d842f99d2eec3b896c5b47` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:09:25.204928 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04dae9f-bb1e-426d-82e3-e490f31a549f admin admin] RBAC: Authorizing `identity:delete_project(project_id=bcf6c738f8d842f99d2eec3b896c5b47)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:09:25.206869 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04dae9f-bb1e-426d-82e3-e490f31a549f admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:09:25.343561 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e04dae9f-bb1e-426d-82e3-e490f31a549f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bcf6c738f8d842f99d2eec3b896c5b47', 'request_id': 'req-e04dae9f-bb1e-426d-82e3-e490f31a549f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:09:25.351257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e04dae9f-bb1e-426d-82e3-e490f31a549f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bcf6c738f8d842f99d2eec3b896c5b47', 'request_id': 'req-e04dae9f-bb1e-426d-82e3-e490f31a549f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:09:25.362471 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e04dae9f-bb1e-426d-82e3-e490f31a549f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bcf6c738f8d842f99d2eec3b896c5b47', 'request_id': 'req-e04dae9f-bb1e-426d-82e3-e490f31a549f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:09:25.453798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e04dae9f-bb1e-426d-82e3-e490f31a549f admin admin] The token cache is being invalidate because project bcf6c738f8d842f99d2eec3b896c5b47 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:09:25.454088 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e04dae9f-bb1e-426d-82e3-e490f31a549f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:09:25.457715 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e04dae9f-bb1e-426d-82e3-e490f31a549f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:09:25.457715 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1113/2221] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:09:25 2022] DELETE /identity/v3/projects/bcf6c738f8d842f99d2eec3b896c5b47 => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:09:31.402652 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6423aadf-11d2-457b-bc08-a28154fd21f7 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:09:31.405550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6423aadf-11d2-457b-bc08-a28154fd21f7 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:09:31.405987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6423aadf-11d2-457b-bc08-a28154fd21f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:09:31.451614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-6423aadf-11d2-457b-bc08-a28154fd21f7 None None] MFA Rules not processed for user `bd13df3a2d0e4182bbd86dab7035c371`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:09:31.452844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6423aadf-11d2-457b-bc08-a28154fd21f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:09:31.527991 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1109/2222] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:09:31 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:09:35.449948 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=57,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:09:41.520830 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=81,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:09:45.827246 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3d1eea-cbe4-48ed-84af-1aff7c055ca1 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:09:45.827892 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3d1eea-cbe4-48ed-84af-1aff7c055ca1 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:09:45.827992 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3d1eea-cbe4-48ed-84af-1aff7c055ca1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:09:45.849194 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-5d3d1eea-cbe4-48ed-84af-1aff7c055ca1 None None] MFA Rules not processed for user `f5e4723258c64ce5885154dded2d3d43`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:09:45.850494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-5d3d1eea-cbe4-48ed-84af-1aff7c055ca1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:09:45.907575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1114/2223] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:09:45 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:09:46.430756 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6423aadf-11d2-457b-bc08-a28154fd21f7 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:09:46.432277 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6423aadf-11d2-457b-bc08-a28154fd21f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:09:46.491336 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6423aadf-11d2-457b-bc08-a28154fd21f7 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:09:46.494904 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e14b8c6-5f8f-4b06-99f9-d7709051fd19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:09:46.498169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e14b8c6-5f8f-4b06-99f9-d7709051fd19 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:09:46.498169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e14b8c6-5f8f-4b06-99f9-d7709051fd19 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:09:46.498169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e14b8c6-5f8f-4b06-99f9-d7709051fd19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:09:46.498169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-5e14b8c6-5f8f-4b06-99f9-d7709051fd19 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:09:46.561593 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e14b8c6-5f8f-4b06-99f9-d7709051fd19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:09:46.562936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e14b8c6-5f8f-4b06-99f9-d7709051fd19 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:09:46.572827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1110/2224] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:09:46 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:09:55.900887 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=25 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:09:55.963176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3d1eea-cbe4-48ed-84af-1aff7c055ca1 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:09:55.973161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3d1eea-cbe4-48ed-84af-1aff7c055ca1 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:09:55.977698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407b905c-7095-4617-8b4d-cde64becee56 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:09:55.978312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407b905c-7095-4617-8b4d-cde64becee56 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:09:55.978593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407b905c-7095-4617-8b4d-cde64becee56 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:09:55.978868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407b905c-7095-4617-8b4d-cde64becee56 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:09:55.980618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-407b905c-7095-4617-8b4d-cde64becee56 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:09:56.054569 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407b905c-7095-4617-8b4d-cde64becee56 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:09:56.056021 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407b905c-7095-4617-8b4d-cde64becee56 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:09:56.066805 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1115/2225] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:09:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:09:56.366496 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e14b8c6-5f8f-4b06-99f9-d7709051fd19 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:09:56.367078 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-5e14b8c6-5f8f-4b06-99f9-d7709051fd19 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:09:56.378041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e14b8c6-5f8f-4b06-99f9-d7709051fd19 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:09:56.382424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af1106d-bb1f-4eac-ad80-73894f192da9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:09:56.383032 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af1106d-bb1f-4eac-ad80-73894f192da9 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:09:56.383317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af1106d-bb1f-4eac-ad80-73894f192da9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:09:56.383689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af1106d-bb1f-4eac-ad80-73894f192da9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:09:56.385531 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-1af1106d-bb1f-4eac-ad80-73894f192da9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:09:56.399428 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af1106d-bb1f-4eac-ad80-73894f192da9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:09:56.400900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af1106d-bb1f-4eac-ad80-73894f192da9 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:09:56.411829 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1111/2226] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:09:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:10:00.379732 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407b905c-7095-4617-8b4d-cde64becee56 service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:00.381214 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-407b905c-7095-4617-8b4d-cde64becee56 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:00.459619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407b905c-7095-4617-8b4d-cde64becee56 service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:00.464703 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c043794-6198-4fb5-90c3-e5b769da5321 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:00.465476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c043794-6198-4fb5-90c3-e5b769da5321 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:00.465647 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c043794-6198-4fb5-90c3-e5b769da5321 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:00.465913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c043794-6198-4fb5-90c3-e5b769da5321 admin admin] PATH_INFO: `/v3/users/77de7bb68f3c45d680c8a78dc40df07b` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:00.467052 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c043794-6198-4fb5-90c3-e5b769da5321 admin admin] RBAC: Authorizing `identity:delete_user(user_id=77de7bb68f3c45d680c8a78dc40df07b)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:00.469057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c043794-6198-4fb5-90c3-e5b769da5321 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:00.662521 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4c043794-6198-4fb5-90c3-e5b769da5321 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '77de7bb68f3c45d680c8a78dc40df07b'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:00.726149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb29a628-885c-4b9a-ac71-a4c6a175a69b None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:00.726662 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb29a628-885c-4b9a-ac71-a4c6a175a69b None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:00.726785 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb29a628-885c-4b9a-ac71-a4c6a175a69b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:00.728601 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4c043794-6198-4fb5-90c3-e5b769da5321 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '77de7bb68f3c45d680c8a78dc40df07b'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:00.743226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4c043794-6198-4fb5-90c3-e5b769da5321 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '77de7bb68f3c45d680c8a78dc40df07b'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:00.744116 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4c043794-6198-4fb5-90c3-e5b769da5321 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '77de7bb68f3c45d680c8a78dc40df07b'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:00.746654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-bb29a628-885c-4b9a-ac71-a4c6a175a69b None None] MFA Rules not processed for user `fbcd4c19e766446188e3b3f41a44e2e4`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:10:00.747871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-bb29a628-885c-4b9a-ac71-a4c6a175a69b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:00.751947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4c043794-6198-4fb5-90c3-e5b769da5321 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '77de7bb68f3c45d680c8a78dc40df07b'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:00.758452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1116/2227] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:10:00 2022] DELETE /identity/v3/users/77de7bb68f3c45d680c8a78dc40df07b => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:10:00.806334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1112/2228] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:10:00 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:10:01.104845 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c043794-6198-4fb5-90c3-e5b769da5321 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:01.104845 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4c043794-6198-4fb5-90c3-e5b769da5321 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:01.110927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:10:01.194950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c043794-6198-4fb5-90c3-e5b769da5321 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:01.199932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ca71df-153b-4d10-b306-d527755c1db8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:01.200570 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ca71df-153b-4d10-b306-d527755c1db8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:01.201220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ca71df-153b-4d10-b306-d527755c1db8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:01.205287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ca71df-153b-4d10-b306-d527755c1db8 admin admin] PATH_INFO: `/v3/projects/6071c3a795ba4aa098e3af8cde306d58` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:01.206632 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ca71df-153b-4d10-b306-d527755c1db8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6071c3a795ba4aa098e3af8cde306d58)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:01.208825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ca71df-153b-4d10-b306-d527755c1db8 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:01.337231 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-21ca71df-153b-4d10-b306-d527755c1db8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6071c3a795ba4aa098e3af8cde306d58', 'request_id': 'req-21ca71df-153b-4d10-b306-d527755c1db8'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:01.345002 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-21ca71df-153b-4d10-b306-d527755c1db8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6071c3a795ba4aa098e3af8cde306d58', 'request_id': 'req-21ca71df-153b-4d10-b306-d527755c1db8'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:01.354196 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-21ca71df-153b-4d10-b306-d527755c1db8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6071c3a795ba4aa098e3af8cde306d58', 'request_id': 'req-21ca71df-153b-4d10-b306-d527755c1db8'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:01.403145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-21ca71df-153b-4d10-b306-d527755c1db8 admin admin] The token cache is being invalidate because project 6071c3a795ba4aa098e3af8cde306d58 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:10:01.403803 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-21ca71df-153b-4d10-b306-d527755c1db8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:01.404321 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-21ca71df-153b-4d10-b306-d527755c1db8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:01.406129 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1117/2229] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:10:01 2022] DELETE /identity/v3/projects/6071c3a795ba4aa098e3af8cde306d58 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:10:10.798715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=81 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:10:11.432740 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=49,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:10:14.681638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1409082c-dfad-4d95-93d8-243d8abf3f80 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:14.682108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1409082c-dfad-4d95-93d8-243d8abf3f80 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:14.682214 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1409082c-dfad-4d95-93d8-243d8abf3f80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:14.703553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-1409082c-dfad-4d95-93d8-243d8abf3f80 None None] MFA Rules not processed for user `44993cd93360442bb0afb28778ee83bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:10:14.706736 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-1409082c-dfad-4d95-93d8-243d8abf3f80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:14.767524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1113/2230] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:10:14 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:10:14.780312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ca71df-153b-4d10-b306-d527755c1db8 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:14.782106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-21ca71df-153b-4d10-b306-d527755c1db8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:14.857689 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ca71df-153b-4d10-b306-d527755c1db8 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:14.863851 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce9d306-f566-4d9b-9696-b3754ba7e1db service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:14.864476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce9d306-f566-4d9b-9696-b3754ba7e1db service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:14.864753 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce9d306-f566-4d9b-9696-b3754ba7e1db service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:14.865030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce9d306-f566-4d9b-9696-b3754ba7e1db service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:14.866935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9ce9d306-f566-4d9b-9696-b3754ba7e1db service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:14.877900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce9d306-f566-4d9b-9696-b3754ba7e1db service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:14.879681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce9d306-f566-4d9b-9696-b3754ba7e1db service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:14.890024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1118/2231] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:10:14 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:10:15.584470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1409082c-dfad-4d95-93d8-243d8abf3f80 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:15.586098 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-1409082c-dfad-4d95-93d8-243d8abf3f80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:15.662690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1409082c-dfad-4d95-93d8-243d8abf3f80 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:15.668420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e32e9c-7d08-4bd4-8f91-310609ebe737 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b29f444d56ec49308f7fef16f4d1eea5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:15.669029 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e32e9c-7d08-4bd4-8f91-310609ebe737 service placement] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:15.669366 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e32e9c-7d08-4bd4-8f91-310609ebe737 service placement] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:15.669639 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e32e9c-7d08-4bd4-8f91-310609ebe737 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:15.671359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e2e32e9c-7d08-4bd4-8f91-310609ebe737 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:15.683137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e32e9c-7d08-4bd4-8f91-310609ebe737 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:15.684722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e32e9c-7d08-4bd4-8f91-310609ebe737 service placement] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:15.694436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1114/2232] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:10:15 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:10:15.809264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce9d306-f566-4d9b-9696-b3754ba7e1db service nova] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:15.820579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce9d306-f566-4d9b-9696-b3754ba7e1db service nova] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:15.826239 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22ccecf-c945-4b18-ab0d-d64e8589cd6d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:15.826838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22ccecf-c945-4b18-ab0d-d64e8589cd6d service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:15.827150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22ccecf-c945-4b18-ab0d-d64e8589cd6d service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:15.827420 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22ccecf-c945-4b18-ab0d-d64e8589cd6d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:15.829629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-f22ccecf-c945-4b18-ab0d-d64e8589cd6d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:15.890932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22ccecf-c945-4b18-ab0d-d64e8589cd6d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:15.892483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22ccecf-c945-4b18-ab0d-d64e8589cd6d service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:15.902372 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1119/2233] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:10:15 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:10:22.654638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e32e9c-7d08-4bd4-8f91-310609ebe737 service placement] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:22.656241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e2e32e9c-7d08-4bd4-8f91-310609ebe737 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:22.717485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e32e9c-7d08-4bd4-8f91-310609ebe737 service placement] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:22.721107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba46c442-2e26-4ed6-a031-1869ab700359 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:22.721733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba46c442-2e26-4ed6-a031-1869ab700359 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:22.722008 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba46c442-2e26-4ed6-a031-1869ab700359 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:22.722274 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba46c442-2e26-4ed6-a031-1869ab700359 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:22.723995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ba46c442-2e26-4ed6-a031-1869ab700359 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:22.783503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:10:22.792382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba46c442-2e26-4ed6-a031-1869ab700359 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:22.794169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba46c442-2e26-4ed6-a031-1869ab700359 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:22.805303 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1115/2234] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:10:22 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:10:25.897294 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=62 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:10:29.647106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22ccecf-c945-4b18-ab0d-d64e8589cd6d service nova] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:29.648594 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-f22ccecf-c945-4b18-ab0d-d64e8589cd6d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:29.709709 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22ccecf-c945-4b18-ab0d-d64e8589cd6d service nova] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:29.713514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25da1823-6eb5-4bbc-96fe-4f11fc03a2c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:29.714116 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25da1823-6eb5-4bbc-96fe-4f11fc03a2c4 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:29.714396 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25da1823-6eb5-4bbc-96fe-4f11fc03a2c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:29.714670 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25da1823-6eb5-4bbc-96fe-4f11fc03a2c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:29.716480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-25da1823-6eb5-4bbc-96fe-4f11fc03a2c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:29.733048 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25da1823-6eb5-4bbc-96fe-4f11fc03a2c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:29.734528 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25da1823-6eb5-4bbc-96fe-4f11fc03a2c4 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:29.742969 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1120/2235] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:10:29 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:10:32.832911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=5 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:10:39.739546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=26 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:10:40.775351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba46c442-2e26-4ed6-a031-1869ab700359 service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:40.777535 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ba46c442-2e26-4ed6-a031-1869ab700359 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:40.793881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba46c442-2e26-4ed6-a031-1869ab700359 service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:40.799621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28acd62d-6dae-455a-b868-0c088451bf59 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:40.800254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28acd62d-6dae-455a-b868-0c088451bf59 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:40.800516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28acd62d-6dae-455a-b868-0c088451bf59 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:40.800875 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28acd62d-6dae-455a-b868-0c088451bf59 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:40.802553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-28acd62d-6dae-455a-b868-0c088451bf59 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:40.815626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28acd62d-6dae-455a-b868-0c088451bf59 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:40.817150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28acd62d-6dae-455a-b868-0c088451bf59 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:40.828236 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1116/2236] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:10:40 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:10:48.206131 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25da1823-6eb5-4bbc-96fe-4f11fc03a2c4 service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:48.218951 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25da1823-6eb5-4bbc-96fe-4f11fc03a2c4 service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:48.224744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50542f4-814d-48ed-813c-330a726d152d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:48.225351 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50542f4-814d-48ed-813c-330a726d152d service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:48.225639 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50542f4-814d-48ed-813c-330a726d152d service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:48.225906 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50542f4-814d-48ed-813c-330a726d152d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:48.227594 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c50542f4-814d-48ed-813c-330a726d152d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:48.285519 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50542f4-814d-48ed-813c-330a726d152d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:48.287092 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50542f4-814d-48ed-813c-330a726d152d service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:48.296602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1121/2237] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:10:48 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:10:48.422337 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28acd62d-6dae-455a-b868-0c088451bf59 service nova] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:48.425444 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-28acd62d-6dae-455a-b868-0c088451bf59 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:48.436240 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28acd62d-6dae-455a-b868-0c088451bf59 service nova] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:48.439903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f52a74b-35ae-4d5a-b7d5-2a926802163a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:48.440507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f52a74b-35ae-4d5a-b7d5-2a926802163a service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:48.440787 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f52a74b-35ae-4d5a-b7d5-2a926802163a service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:48.441064 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f52a74b-35ae-4d5a-b7d5-2a926802163a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:48.442882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-0f52a74b-35ae-4d5a-b7d5-2a926802163a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:48.512712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f52a74b-35ae-4d5a-b7d5-2a926802163a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:48.514192 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f52a74b-35ae-4d5a-b7d5-2a926802163a service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:48.525222 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1117/2238] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:10:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:10:52.196923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50542f4-814d-48ed-813c-330a726d152d service nova] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:52.211457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50542f4-814d-48ed-813c-330a726d152d service nova] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:52.214760 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67e0e7a-0194-4dd9-959b-b292d94dd44f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b29f444d56ec49308f7fef16f4d1eea5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:52.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67e0e7a-0194-4dd9-959b-b292d94dd44f service placement] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:52.215637 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67e0e7a-0194-4dd9-959b-b292d94dd44f service placement] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:52.215911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67e0e7a-0194-4dd9-959b-b292d94dd44f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:52.217909 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c67e0e7a-0194-4dd9-959b-b292d94dd44f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:52.230750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67e0e7a-0194-4dd9-959b-b292d94dd44f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:52.232354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67e0e7a-0194-4dd9-959b-b292d94dd44f service placement] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:52.242974 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1122/2239] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:10:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:10:52.311044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f52a74b-35ae-4d5a-b7d5-2a926802163a service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:52.312508 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-0f52a74b-35ae-4d5a-b7d5-2a926802163a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:52.325464 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f52a74b-35ae-4d5a-b7d5-2a926802163a service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:52.330427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909c4721-e867-4c16-a89b-c184a692f717 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:52.331063 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909c4721-e867-4c16-a89b-c184a692f717 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:52.331345 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909c4721-e867-4c16-a89b-c184a692f717 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:52.331621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909c4721-e867-4c16-a89b-c184a692f717 admin admin] PATH_INFO: `/v3/users/bd13df3a2d0e4182bbd86dab7035c371` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:52.332799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909c4721-e867-4c16-a89b-c184a692f717 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bd13df3a2d0e4182bbd86dab7035c371)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:52.334773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909c4721-e867-4c16-a89b-c184a692f717 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:52.530858 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-909c4721-e867-4c16-a89b-c184a692f717 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bd13df3a2d0e4182bbd86dab7035c371'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:52.531517 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-909c4721-e867-4c16-a89b-c184a692f717 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bd13df3a2d0e4182bbd86dab7035c371'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:52.648088 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-909c4721-e867-4c16-a89b-c184a692f717 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bd13df3a2d0e4182bbd86dab7035c371'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:52.661229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-909c4721-e867-4c16-a89b-c184a692f717 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bd13df3a2d0e4182bbd86dab7035c371'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:52.661817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-909c4721-e867-4c16-a89b-c184a692f717 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bd13df3a2d0e4182bbd86dab7035c371'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:52.674898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1118/2240] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:10:52 2022] DELETE /identity/v3/users/bd13df3a2d0e4182bbd86dab7035c371 => generated 0 bytes in 364 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:10:52.929719 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67e0e7a-0194-4dd9-959b-b292d94dd44f service placement] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:52.941255 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67e0e7a-0194-4dd9-959b-b292d94dd44f service placement] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:52.998475 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b806899b-93a3-4c97-af2b-05d94b8fd38b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:52.999158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b806899b-93a3-4c97-af2b-05d94b8fd38b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:52.999441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b806899b-93a3-4c97-af2b-05d94b8fd38b admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:52.999714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b806899b-93a3-4c97-af2b-05d94b8fd38b admin admin] PATH_INFO: `/v3/projects/db72b63281384bd883b04d05bcc35677` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:53.000830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b806899b-93a3-4c97-af2b-05d94b8fd38b admin admin] RBAC: Authorizing `identity:delete_project(project_id=db72b63281384bd883b04d05bcc35677)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:53.002782 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b806899b-93a3-4c97-af2b-05d94b8fd38b admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:53.099854 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b806899b-93a3-4c97-af2b-05d94b8fd38b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db72b63281384bd883b04d05bcc35677', 'request_id': 'req-b806899b-93a3-4c97-af2b-05d94b8fd38b'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:53.105303 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b806899b-93a3-4c97-af2b-05d94b8fd38b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db72b63281384bd883b04d05bcc35677', 'request_id': 'req-b806899b-93a3-4c97-af2b-05d94b8fd38b'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:53.111501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b806899b-93a3-4c97-af2b-05d94b8fd38b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db72b63281384bd883b04d05bcc35677', 'request_id': 'req-b806899b-93a3-4c97-af2b-05d94b8fd38b'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:53.195863 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b806899b-93a3-4c97-af2b-05d94b8fd38b admin admin] The token cache is being invalidate because project db72b63281384bd883b04d05bcc35677 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:10:53.196116 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b806899b-93a3-4c97-af2b-05d94b8fd38b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:53.196756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b806899b-93a3-4c97-af2b-05d94b8fd38b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:10:53.198971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1123/2241] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:10:52 2022] DELETE /identity/v3/projects/db72b63281384bd883b04d05bcc35677 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:10:53.208092 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a0ecba-8710-48c4-86c0-8ba70bc033ba None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:53.208092 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a0ecba-8710-48c4-86c0-8ba70bc033ba None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:53.208092 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a0ecba-8710-48c4-86c0-8ba70bc033ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:53.225430 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-62a0ecba-8710-48c4-86c0-8ba70bc033ba None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:10:53.226701 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-62a0ecba-8710-48c4-86c0-8ba70bc033ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:53.276075 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:10:53.289168 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1119/2242] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:10:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:10:53.297658 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd8128f-ce83-43ea-a0c8-05932f238461 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:53.298142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd8128f-ce83-43ea-a0c8-05932f238461 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:53.299397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd8128f-ce83-43ea-a0c8-05932f238461 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:53.329923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9cd8128f-ce83-43ea-a0c8-05932f238461 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:10:53.331725 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9cd8128f-ce83-43ea-a0c8-05932f238461 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:53.347908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1124/2243] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:10:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:10:53.349662 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a0ecba-8710-48c4-86c0-8ba70bc033ba None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:53.360447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a0ecba-8710-48c4-86c0-8ba70bc033ba None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:53.365754 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e4de3b-10f4-4587-bfbd-aad9c62a2550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:53.366370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e4de3b-10f4-4587-bfbd-aad9c62a2550 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:53.366649 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e4de3b-10f4-4587-bfbd-aad9c62a2550 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:53.366917 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e4de3b-10f4-4587-bfbd-aad9c62a2550 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:53.367581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e4de3b-10f4-4587-bfbd-aad9c62a2550 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:10:53.367899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e4de3b-10f4-4587-bfbd-aad9c62a2550 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:53.369660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e4de3b-10f4-4587-bfbd-aad9c62a2550 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:53.383152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1120/2244] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:10:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:10:53.388369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd8128f-ce83-43ea-a0c8-05932f238461 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:53.397933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd8128f-ce83-43ea-a0c8-05932f238461 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:53.403124 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0ad01f-9e01-40a5-b4a7-7d5103ecb3c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:53.403731 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0ad01f-9e01-40a5-b4a7-7d5103ecb3c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:53.404006 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0ad01f-9e01-40a5-b4a7-7d5103ecb3c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:53.404281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0ad01f-9e01-40a5-b4a7-7d5103ecb3c0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:53.404937 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0ad01f-9e01-40a5-b4a7-7d5103ecb3c0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:53.406944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0ad01f-9e01-40a5-b4a7-7d5103ecb3c0 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:53.412872 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:10:53.513024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1125/2245] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:10:53 2022] POST /identity/v3/projects => generated 381 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:10:53.517476 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e4de3b-10f4-4587-bfbd-aad9c62a2550 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:53.527451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e4de3b-10f4-4587-bfbd-aad9c62a2550 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:53.584257 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75092b1e-9567-4cfc-84ec-d4a6b5d59215 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:53.584860 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75092b1e-9567-4cfc-84ec-d4a6b5d59215 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:53.585169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75092b1e-9567-4cfc-84ec-d4a6b5d59215 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:53.585438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75092b1e-9567-4cfc-84ec-d4a6b5d59215 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:53.586052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75092b1e-9567-4cfc-84ec-d4a6b5d59215 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:53.587953 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75092b1e-9567-4cfc-84ec-d4a6b5d59215 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:53.682562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1121/2246] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:10:53 2022] POST /identity/v3/users => generated 291 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:10:53.687341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0ad01f-9e01-40a5-b4a7-7d5103ecb3c0 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:53.697589 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0ad01f-9e01-40a5-b4a7-7d5103ecb3c0 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:53.702498 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d080f0-517e-4031-b3bf-61f57910cdbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:53.703087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d080f0-517e-4031-b3bf-61f57910cdbe admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:53.703368 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d080f0-517e-4031-b3bf-61f57910cdbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:53.703637 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d080f0-517e-4031-b3bf-61f57910cdbe admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:53.704273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d080f0-517e-4031-b3bf-61f57910cdbe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:53.705920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d080f0-517e-4031-b3bf-61f57910cdbe admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:53.715735 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1126/2247] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:10:53 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:10:53.719348 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75092b1e-9567-4cfc-84ec-d4a6b5d59215 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:53.729059 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75092b1e-9567-4cfc-84ec-d4a6b5d59215 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:53.734165 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4dd8ee-9d66-4c0a-b4a4-edb54dcddd2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:53.734750 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4dd8ee-9d66-4c0a-b4a4-edb54dcddd2e admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:53.735030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4dd8ee-9d66-4c0a-b4a4-edb54dcddd2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:53.735304 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4dd8ee-9d66-4c0a-b4a4-edb54dcddd2e admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:53.735951 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4dd8ee-9d66-4c0a-b4a4-edb54dcddd2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:53.737568 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4dd8ee-9d66-4c0a-b4a4-edb54dcddd2e admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:53.748823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1122/2248] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:10:53 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:10:53.752889 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d080f0-517e-4031-b3bf-61f57910cdbe admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:53.764144 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d080f0-517e-4031-b3bf-61f57910cdbe admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:53.768780 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1526bb2-0fa0-4b7c-899a-d9bdb1dfed81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:53.769460 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1526bb2-0fa0-4b7c-899a-d9bdb1dfed81 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:53.769768 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1526bb2-0fa0-4b7c-899a-d9bdb1dfed81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:53.770045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1526bb2-0fa0-4b7c-899a-d9bdb1dfed81 admin admin] PATH_INFO: `/v3/projects/d04a2e7c3c4743dc8ef7d9a3b0fc922b/users/792be1366c29487bb6ff72181a180d48/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:53.784474 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1526bb2-0fa0-4b7c-899a-d9bdb1dfed81 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d04a2e7c3c4743dc8ef7d9a3b0fc922b, user_id=792be1366c29487bb6ff72181a180d48, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:53.787161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1526bb2-0fa0-4b7c-899a-d9bdb1dfed81 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:53.858257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1127/2249] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:10:53 2022] PUT /identity/v3/projects/d04a2e7c3c4743dc8ef7d9a3b0fc922b/users/792be1366c29487bb6ff72181a180d48/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:10:53.869614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0b1ad1-52b1-4900-a85c-dc14117019ff None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:53.869894 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0b1ad1-52b1-4900-a85c-dc14117019ff None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:53.870817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0b1ad1-52b1-4900-a85c-dc14117019ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:53.900849 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-8b0b1ad1-52b1-4900-a85c-dc14117019ff None None] MFA Rules not processed for user `792be1366c29487bb6ff72181a180d48`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:10:53.902687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8b0b1ad1-52b1-4900-a85c-dc14117019ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:53.971087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1123/2250] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:10:53 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:10:53.975555 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1526bb2-0fa0-4b7c-899a-d9bdb1dfed81 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:54.039313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1526bb2-0fa0-4b7c-899a-d9bdb1dfed81 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:54.044994 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f60ab73-25e2-45a4-a7e4-f9ac653bf2d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:54.045726 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f60ab73-25e2-45a4-a7e4-f9ac653bf2d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:54.046240 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f60ab73-25e2-45a4-a7e4-f9ac653bf2d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:54.046789 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f60ab73-25e2-45a4-a7e4-f9ac653bf2d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:54.047983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f60ab73-25e2-45a4-a7e4-f9ac653bf2d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:54.050182 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f60ab73-25e2-45a4-a7e4-f9ac653bf2d9 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:54.135269 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1128/2251] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:10:53 2022] POST /identity/v3/projects => generated 379 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:10:54.140316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0b1ad1-52b1-4900-a85c-dc14117019ff None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:54.151300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0b1ad1-52b1-4900-a85c-dc14117019ff None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:54.196453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:10:54.208728 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337c7e2b-8e1b-43f6-bb9c-e806e911313e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:54.209972 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337c7e2b-8e1b-43f6-bb9c-e806e911313e admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:54.209972 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337c7e2b-8e1b-43f6-bb9c-e806e911313e admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:54.209972 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337c7e2b-8e1b-43f6-bb9c-e806e911313e admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:54.210499 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337c7e2b-8e1b-43f6-bb9c-e806e911313e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:54.212396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337c7e2b-8e1b-43f6-bb9c-e806e911313e admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:54.299702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1124/2252] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:10:54 2022] POST /identity/v3/users => generated 296 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:10:54.303618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f60ab73-25e2-45a4-a7e4-f9ac653bf2d9 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:54.314774 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f60ab73-25e2-45a4-a7e4-f9ac653bf2d9 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:54.319386 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a32432-1fd7-404b-bed8-f0a6557ed537 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:54.319974 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a32432-1fd7-404b-bed8-f0a6557ed537 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:54.320257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a32432-1fd7-404b-bed8-f0a6557ed537 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:54.320532 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a32432-1fd7-404b-bed8-f0a6557ed537 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:54.321223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a32432-1fd7-404b-bed8-f0a6557ed537 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:54.322831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a32432-1fd7-404b-bed8-f0a6557ed537 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:54.337087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1129/2253] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:10:54 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:10:54.342712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337c7e2b-8e1b-43f6-bb9c-e806e911313e admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:54.353481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337c7e2b-8e1b-43f6-bb9c-e806e911313e admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:54.359173 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06678d8-3eeb-44af-823a-aec556ceef85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:54.359829 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06678d8-3eeb-44af-823a-aec556ceef85 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:54.360107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06678d8-3eeb-44af-823a-aec556ceef85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:54.360379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06678d8-3eeb-44af-823a-aec556ceef85 admin admin] PATH_INFO: `/v3/projects/ecc3867e695f49be92a928e0674222e2/users/126c4a172fe343f5aa00b656d47cab2c/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:54.376028 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06678d8-3eeb-44af-823a-aec556ceef85 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ecc3867e695f49be92a928e0674222e2, user_id=126c4a172fe343f5aa00b656d47cab2c, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:54.380391 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06678d8-3eeb-44af-823a-aec556ceef85 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:54.467030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1125/2254] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:10:54 2022] PUT /identity/v3/projects/ecc3867e695f49be92a928e0674222e2/users/126c4a172fe343f5aa00b656d47cab2c/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:10:54.471012 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a32432-1fd7-404b-bed8-f0a6557ed537 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:54.482460 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a32432-1fd7-404b-bed8-f0a6557ed537 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:54.540001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6097e4-a0cb-44c9-844c-4feb09cb24be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:54.540598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6097e4-a0cb-44c9-844c-4feb09cb24be admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:54.540879 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6097e4-a0cb-44c9-844c-4feb09cb24be admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:54.541188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6097e4-a0cb-44c9-844c-4feb09cb24be admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:54.541830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6097e4-a0cb-44c9-844c-4feb09cb24be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:54.543984 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6097e4-a0cb-44c9-844c-4feb09cb24be admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:54.553682 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1130/2255] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:10:54 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:10:54.558416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06678d8-3eeb-44af-823a-aec556ceef85 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:54.568500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06678d8-3eeb-44af-823a-aec556ceef85 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:54.572949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0e48ba-19b3-4306-bdd3-54800534e4bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:54.573726 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0e48ba-19b3-4306-bdd3-54800534e4bc admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:54.573930 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0e48ba-19b3-4306-bdd3-54800534e4bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:54.574201 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0e48ba-19b3-4306-bdd3-54800534e4bc admin admin] PATH_INFO: `/v3/projects/ecc3867e695f49be92a928e0674222e2/users/126c4a172fe343f5aa00b656d47cab2c/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:54.576087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0e48ba-19b3-4306-bdd3-54800534e4bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=ecc3867e695f49be92a928e0674222e2, user_id=126c4a172fe343f5aa00b656d47cab2c, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:54.578722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0e48ba-19b3-4306-bdd3-54800534e4bc admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:54.588501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1126/2256] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:10:54 2022] PUT /identity/v3/projects/ecc3867e695f49be92a928e0674222e2/users/126c4a172fe343f5aa00b656d47cab2c/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:10:54.598333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dac009a-eb7f-4976-996a-a09db9e6a81a None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:54.598625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dac009a-eb7f-4976-996a-a09db9e6a81a None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:54.598929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dac009a-eb7f-4976-996a-a09db9e6a81a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:54.627451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9dac009a-eb7f-4976-996a-a09db9e6a81a None None] MFA Rules not processed for user `126c4a172fe343f5aa00b656d47cab2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:10:54.628655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9dac009a-eb7f-4976-996a-a09db9e6a81a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:54.637725 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:10:54.705433 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1131/2257] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:10:54 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:10:54.716539 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0e48ba-19b3-4306-bdd3-54800534e4bc admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:54.718309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-bc0e48ba-19b3-4306-bdd3-54800534e4bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:10:54.780085 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0e48ba-19b3-4306-bdd3-54800534e4bc admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:54.783714 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a95ad7b-119f-4cc6-a3db-a419dca2155b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:54.784426 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a95ad7b-119f-4cc6-a3db-a419dca2155b service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:54.785012 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a95ad7b-119f-4cc6-a3db-a419dca2155b service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:54.785316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a95ad7b-119f-4cc6-a3db-a419dca2155b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:54.794835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a95ad7b-119f-4cc6-a3db-a419dca2155b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:54.796291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a95ad7b-119f-4cc6-a3db-a419dca2155b service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:54.805544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1127/2258] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:10:54 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:10:55.013081 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dac009a-eb7f-4976-996a-a09db9e6a81a None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:10:55.022915 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dac009a-eb7f-4976-996a-a09db9e6a81a None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:10:55.027196 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbafd41b-7c75-4271-a8bd-28c1a90e1312 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:10:55.027792 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbafd41b-7c75-4271-a8bd-28c1a90e1312 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:10:55.028069 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbafd41b-7c75-4271-a8bd-28c1a90e1312 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:10:55.028338 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbafd41b-7c75-4271-a8bd-28c1a90e1312 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:10:55.084609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbafd41b-7c75-4271-a8bd-28c1a90e1312 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:10:55.085994 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbafd41b-7c75-4271-a8bd-28c1a90e1312 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:10:55.095441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1132/2259] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:10:55 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:11:04.801465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=48,keystone:INSERT=5 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:05.091279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=52 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:06.904478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a95ad7b-119f-4cc6-a3db-a419dca2155b service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:06.976080 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a95ad7b-119f-4cc6-a3db-a419dca2155b service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:06.980999 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37f113d-ffeb-4cfb-b1fe-fe6e5b01d7d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:06.981675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37f113d-ffeb-4cfb-b1fe-fe6e5b01d7d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:06.981959 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37f113d-ffeb-4cfb-b1fe-fe6e5b01d7d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:06.982231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37f113d-ffeb-4cfb-b1fe-fe6e5b01d7d8 admin admin] PATH_INFO: `/v3/users/792be1366c29487bb6ff72181a180d48` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:06.984501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37f113d-ffeb-4cfb-b1fe-fe6e5b01d7d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=792be1366c29487bb6ff72181a180d48)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:06.987297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37f113d-ffeb-4cfb-b1fe-fe6e5b01d7d8 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:07.240668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b37f113d-ffeb-4cfb-b1fe-fe6e5b01d7d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '792be1366c29487bb6ff72181a180d48'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:07.241388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b37f113d-ffeb-4cfb-b1fe-fe6e5b01d7d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '792be1366c29487bb6ff72181a180d48'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:07.334661 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b37f113d-ffeb-4cfb-b1fe-fe6e5b01d7d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '792be1366c29487bb6ff72181a180d48'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:07.348242 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b37f113d-ffeb-4cfb-b1fe-fe6e5b01d7d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '792be1366c29487bb6ff72181a180d48'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:07.349426 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b37f113d-ffeb-4cfb-b1fe-fe6e5b01d7d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '792be1366c29487bb6ff72181a180d48'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:07.362729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1128/2260] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:11:06 2022] DELETE /identity/v3/users/792be1366c29487bb6ff72181a180d48 => generated 0 bytes in 459 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:07.368151 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6710ee-ddb3-4a38-8c68-01478861ff37 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:07.368624 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6710ee-ddb3-4a38-8c68-01478861ff37 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:07.368718 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6710ee-ddb3-4a38-8c68-01478861ff37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:07.388571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-5f6710ee-ddb3-4a38-8c68-01478861ff37 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:07.389829 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-5f6710ee-ddb3-4a38-8c68-01478861ff37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:07.450022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1133/2261] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:07.459895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37f113d-ffeb-4cfb-b1fe-fe6e5b01d7d8 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:07.461459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b37f113d-ffeb-4cfb-b1fe-fe6e5b01d7d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:07.520308 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37f113d-ffeb-4cfb-b1fe-fe6e5b01d7d8 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:07.524432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae198e18-e709-4e34-a028-22be320de60d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:07.525222 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae198e18-e709-4e34-a028-22be320de60d service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:07.525505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae198e18-e709-4e34-a028-22be320de60d service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:07.525780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae198e18-e709-4e34-a028-22be320de60d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:07.536274 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae198e18-e709-4e34-a028-22be320de60d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:07.537752 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae198e18-e709-4e34-a028-22be320de60d service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:07.547951 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1129/2262] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:11:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:11:07.936108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6710ee-ddb3-4a38-8c68-01478861ff37 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:07.937832 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-5f6710ee-ddb3-4a38-8c68-01478861ff37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:07.950504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f6710ee-ddb3-4a38-8c68-01478861ff37 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:07.955608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edecd82-7ab4-4cc5-af79-7ba1efa64797 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:07.956362 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edecd82-7ab4-4cc5-af79-7ba1efa64797 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:07.956800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edecd82-7ab4-4cc5-af79-7ba1efa64797 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:07.957090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edecd82-7ab4-4cc5-af79-7ba1efa64797 admin admin] PATH_INFO: `/v3/projects/d04a2e7c3c4743dc8ef7d9a3b0fc922b` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:07.958981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edecd82-7ab4-4cc5-af79-7ba1efa64797 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d04a2e7c3c4743dc8ef7d9a3b0fc922b)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:07.961368 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edecd82-7ab4-4cc5-af79-7ba1efa64797 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:08.052506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-1edecd82-7ab4-4cc5-af79-7ba1efa64797 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd04a2e7c3c4743dc8ef7d9a3b0fc922b', 'request_id': 'req-1edecd82-7ab4-4cc5-af79-7ba1efa64797'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:08.065061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-1edecd82-7ab4-4cc5-af79-7ba1efa64797 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd04a2e7c3c4743dc8ef7d9a3b0fc922b', 'request_id': 'req-1edecd82-7ab4-4cc5-af79-7ba1efa64797'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:08.071876 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-1edecd82-7ab4-4cc5-af79-7ba1efa64797 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd04a2e7c3c4743dc8ef7d9a3b0fc922b', 'request_id': 'req-1edecd82-7ab4-4cc5-af79-7ba1efa64797'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:08.133727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-1edecd82-7ab4-4cc5-af79-7ba1efa64797 admin admin] The token cache is being invalidate because project d04a2e7c3c4743dc8ef7d9a3b0fc922b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:11:08.134020 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-1edecd82-7ab4-4cc5-af79-7ba1efa64797 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:08.134648 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-1edecd82-7ab4-4cc5-af79-7ba1efa64797 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:08.136135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1134/2263] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:11:07 2022] DELETE /identity/v3/projects/d04a2e7c3c4743dc8ef7d9a3b0fc922b => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:08.140851 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae198e18-e709-4e34-a028-22be320de60d service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:08.151609 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae198e18-e709-4e34-a028-22be320de60d service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:08.174455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:08.219284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6133003e-f1ac-4ab3-8358-25fb3ad86e4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:08.220353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6133003e-f1ac-4ab3-8358-25fb3ad86e4c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:08.220853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6133003e-f1ac-4ab3-8358-25fb3ad86e4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:08.221499 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6133003e-f1ac-4ab3-8358-25fb3ad86e4c admin admin] PATH_INFO: `/v3/users/126c4a172fe343f5aa00b656d47cab2c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:08.223216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6133003e-f1ac-4ab3-8358-25fb3ad86e4c admin admin] RBAC: Authorizing `identity:delete_user(user_id=126c4a172fe343f5aa00b656d47cab2c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:08.225367 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6133003e-f1ac-4ab3-8358-25fb3ad86e4c admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:08.397046 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-6133003e-f1ac-4ab3-8358-25fb3ad86e4c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '126c4a172fe343f5aa00b656d47cab2c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:08.397764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-6133003e-f1ac-4ab3-8358-25fb3ad86e4c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '126c4a172fe343f5aa00b656d47cab2c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:08.448496 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-6133003e-f1ac-4ab3-8358-25fb3ad86e4c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '126c4a172fe343f5aa00b656d47cab2c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:08.472835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-6133003e-f1ac-4ab3-8358-25fb3ad86e4c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '126c4a172fe343f5aa00b656d47cab2c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:08.473800 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-6133003e-f1ac-4ab3-8358-25fb3ad86e4c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '126c4a172fe343f5aa00b656d47cab2c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:08.491444 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1130/2264] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:11:08 2022] DELETE /identity/v3/users/126c4a172fe343f5aa00b656d47cab2c => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:08.957258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edecd82-7ab4-4cc5-af79-7ba1efa64797 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:08.958791 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-1edecd82-7ab4-4cc5-af79-7ba1efa64797 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:08.979912 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edecd82-7ab4-4cc5-af79-7ba1efa64797 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:09.041838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb120610-d72b-447b-988b-514802a7947d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:09.042474 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb120610-d72b-447b-988b-514802a7947d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:09.042755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb120610-d72b-447b-988b-514802a7947d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:09.043066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb120610-d72b-447b-988b-514802a7947d admin admin] PATH_INFO: `/v3/projects/ecc3867e695f49be92a928e0674222e2` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:09.044257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb120610-d72b-447b-988b-514802a7947d admin admin] RBAC: Authorizing `identity:delete_project(project_id=ecc3867e695f49be92a928e0674222e2)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:09.046506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb120610-d72b-447b-988b-514802a7947d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:09.147243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:09.163641 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cb120610-d72b-447b-988b-514802a7947d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ecc3867e695f49be92a928e0674222e2', 'request_id': 'req-cb120610-d72b-447b-988b-514802a7947d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:09.168992 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cb120610-d72b-447b-988b-514802a7947d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ecc3867e695f49be92a928e0674222e2', 'request_id': 'req-cb120610-d72b-447b-988b-514802a7947d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:09.175777 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cb120610-d72b-447b-988b-514802a7947d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ecc3867e695f49be92a928e0674222e2', 'request_id': 'req-cb120610-d72b-447b-988b-514802a7947d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:09.255070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cb120610-d72b-447b-988b-514802a7947d admin admin] The token cache is being invalidate because project ecc3867e695f49be92a928e0674222e2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:11:09.255369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cb120610-d72b-447b-988b-514802a7947d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:09.256005 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cb120610-d72b-447b-988b-514802a7947d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:09.257943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1135/2265] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:11:08 2022] DELETE /identity/v3/projects/ecc3867e695f49be92a928e0674222e2 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:09.262799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e66187-d5c7-4136-bd06-c71258c7ea56 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:09.263300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e66187-d5c7-4136-bd06-c71258c7ea56 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:09.263517 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e66187-d5c7-4136-bd06-c71258c7ea56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:09.287520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-92e66187-d5c7-4136-bd06-c71258c7ea56 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:09.288764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-92e66187-d5c7-4136-bd06-c71258c7ea56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:09.350614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1131/2266] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:09.360258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3fdacd-4b6b-49c1-ba04-f41c85041bed None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:09.360543 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3fdacd-4b6b-49c1-ba04-f41c85041bed None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:09.360818 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3fdacd-4b6b-49c1-ba04-f41c85041bed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:09.381973 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-2f3fdacd-4b6b-49c1-ba04-f41c85041bed None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:09.383215 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-2f3fdacd-4b6b-49c1-ba04-f41c85041bed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:09.392054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1136/2267] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:09.396595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e66187-d5c7-4136-bd06-c71258c7ea56 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:09.408128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e66187-d5c7-4136-bd06-c71258c7ea56 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:09.414107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4cd5359-f434-4fc0-a942-e9c1aed4e021 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:09.414870 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cd5359-f434-4fc0-a942-e9c1aed4e021 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:09.415157 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cd5359-f434-4fc0-a942-e9c1aed4e021 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:09.415435 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cd5359-f434-4fc0-a942-e9c1aed4e021 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:09.416658 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4cd5359-f434-4fc0-a942-e9c1aed4e021 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:11:09.416658 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4cd5359-f434-4fc0-a942-e9c1aed4e021 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:09.418121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4cd5359-f434-4fc0-a942-e9c1aed4e021 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:09.437425 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1132/2268] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:11:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 01:11:09.438801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3fdacd-4b6b-49c1-ba04-f41c85041bed None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:09.448261 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3fdacd-4b6b-49c1-ba04-f41c85041bed None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:09.454284 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df15f5f4-bbeb-46a4-9600-33002e94033b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:09.455312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df15f5f4-bbeb-46a4-9600-33002e94033b admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:09.455592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df15f5f4-bbeb-46a4-9600-33002e94033b admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:09.455866 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df15f5f4-bbeb-46a4-9600-33002e94033b admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:09.456505 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df15f5f4-bbeb-46a4-9600-33002e94033b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:09.458488 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df15f5f4-bbeb-46a4-9600-33002e94033b admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:09.535758 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1137/2269] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:11:09 2022] POST /identity/v3/projects => generated 379 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:11:09.539966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4cd5359-f434-4fc0-a942-e9c1aed4e021 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:09.550655 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4cd5359-f434-4fc0-a942-e9c1aed4e021 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:09.581212 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:09.613687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8556e4f-3d0f-4963-b03a-60eda5adaf5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:09.614588 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8556e4f-3d0f-4963-b03a-60eda5adaf5f admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:09.614865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8556e4f-3d0f-4963-b03a-60eda5adaf5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:09.615136 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8556e4f-3d0f-4963-b03a-60eda5adaf5f admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:09.615981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8556e4f-3d0f-4963-b03a-60eda5adaf5f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:09.617915 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8556e4f-3d0f-4963-b03a-60eda5adaf5f admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:09.717516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1133/2270] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:11:09 2022] POST /identity/v3/users => generated 290 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:11:09.721753 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df15f5f4-bbeb-46a4-9600-33002e94033b admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:09.732026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df15f5f4-bbeb-46a4-9600-33002e94033b admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:09.737480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b49091d-396e-46b4-ac78-cda7a6765dec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:09.738096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b49091d-396e-46b4-ac78-cda7a6765dec admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:09.738380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b49091d-396e-46b4-ac78-cda7a6765dec admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:09.738654 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b49091d-396e-46b4-ac78-cda7a6765dec admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:09.739546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b49091d-396e-46b4-ac78-cda7a6765dec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:09.741180 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b49091d-396e-46b4-ac78-cda7a6765dec admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:09.751609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1138/2271] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:11:09.755515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8556e4f-3d0f-4963-b03a-60eda5adaf5f admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:09.765339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8556e4f-3d0f-4963-b03a-60eda5adaf5f admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:09.769920 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b35eec-d0bc-460d-845c-f37a72610a9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:09.770490 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b35eec-d0bc-460d-845c-f37a72610a9a admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:09.770766 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b35eec-d0bc-460d-845c-f37a72610a9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:09.771040 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b35eec-d0bc-460d-845c-f37a72610a9a admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:09.771676 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b35eec-d0bc-460d-845c-f37a72610a9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:09.773299 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b35eec-d0bc-460d-845c-f37a72610a9a admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:09.784761 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1134/2272] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:11:09.788762 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b49091d-396e-46b4-ac78-cda7a6765dec admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:09.806693 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b49091d-396e-46b4-ac78-cda7a6765dec admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:09.806693 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f830898-da4a-464a-8f43-9f26ebdf702c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:09.806693 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f830898-da4a-464a-8f43-9f26ebdf702c admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:09.806693 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f830898-da4a-464a-8f43-9f26ebdf702c admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:09.807363 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f830898-da4a-464a-8f43-9f26ebdf702c admin admin] PATH_INFO: `/v3/projects/116334f5d153436bae64e2cb17629671/users/97ec75d3f9cb4396a7e9ec74ff5554fb/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:09.822224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f830898-da4a-464a-8f43-9f26ebdf702c admin admin] RBAC: Authorizing `identity:create_grant(project_id=116334f5d153436bae64e2cb17629671, user_id=97ec75d3f9cb4396a7e9ec74ff5554fb, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:09.824962 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f830898-da4a-464a-8f43-9f26ebdf702c admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:09.899528 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1139/2273] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:11:09 2022] PUT /identity/v3/projects/116334f5d153436bae64e2cb17629671/users/97ec75d3f9cb4396a7e9ec74ff5554fb/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:09.918423 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8971e6-c14c-484a-9f29-f5abef910585 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:09.918755 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8971e6-c14c-484a-9f29-f5abef910585 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:09.918979 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8971e6-c14c-484a-9f29-f5abef910585 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:09.948956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-9c8971e6-c14c-484a-9f29-f5abef910585 None None] MFA Rules not processed for user `97ec75d3f9cb4396a7e9ec74ff5554fb`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:09.950169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-9c8971e6-c14c-484a-9f29-f5abef910585 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:10.016869 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1135/2274] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:09 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:10.021542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f830898-da4a-464a-8f43-9f26ebdf702c admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:10.086828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f830898-da4a-464a-8f43-9f26ebdf702c admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:10.091435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba7eeb7-07eb-4699-b13d-9e05ede65ddd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:10.092276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba7eeb7-07eb-4699-b13d-9e05ede65ddd admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:10.092553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba7eeb7-07eb-4699-b13d-9e05ede65ddd admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:10.092824 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba7eeb7-07eb-4699-b13d-9e05ede65ddd admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:10.093481 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba7eeb7-07eb-4699-b13d-9e05ede65ddd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:10.097697 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba7eeb7-07eb-4699-b13d-9e05ede65ddd admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:10.181950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1140/2275] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:11:10 2022] POST /identity/v3/projects => generated 381 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:11:10.186420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8971e6-c14c-484a-9f29-f5abef910585 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:10.197144 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8971e6-c14c-484a-9f29-f5abef910585 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:10.267803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509253f1-eca8-4546-8ba9-c77ef5a21bfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:10.268414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509253f1-eca8-4546-8ba9-c77ef5a21bfc admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:10.268414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509253f1-eca8-4546-8ba9-c77ef5a21bfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:10.268722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509253f1-eca8-4546-8ba9-c77ef5a21bfc admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:10.269290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509253f1-eca8-4546-8ba9-c77ef5a21bfc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:10.271235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509253f1-eca8-4546-8ba9-c77ef5a21bfc admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:10.354481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1136/2276] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:11:10 2022] POST /identity/v3/users => generated 297 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:11:10.355762 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba7eeb7-07eb-4699-b13d-9e05ede65ddd admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:10.367638 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba7eeb7-07eb-4699-b13d-9e05ede65ddd admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:10.372567 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfb1b9c-71fc-4a6e-8ccf-2577fc16a34c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:10.373825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfb1b9c-71fc-4a6e-8ccf-2577fc16a34c admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:10.374105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfb1b9c-71fc-4a6e-8ccf-2577fc16a34c admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:10.374377 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfb1b9c-71fc-4a6e-8ccf-2577fc16a34c admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:10.375027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfb1b9c-71fc-4a6e-8ccf-2577fc16a34c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:10.376634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfb1b9c-71fc-4a6e-8ccf-2577fc16a34c admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:10.387399 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1141/2277] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:10 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:11:10.392159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509253f1-eca8-4546-8ba9-c77ef5a21bfc admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:10.402103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509253f1-eca8-4546-8ba9-c77ef5a21bfc admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:10.407365 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d9a0f4-d80e-44fd-9ea6-886917959e77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:10.408018 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d9a0f4-d80e-44fd-9ea6-886917959e77 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:10.408291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d9a0f4-d80e-44fd-9ea6-886917959e77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:10.408564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d9a0f4-d80e-44fd-9ea6-886917959e77 admin admin] PATH_INFO: `/v3/projects/db8d12b797ca4982ad0ebbc95384040a/users/2f37f72c6cdf48ad9415b6b9297b9f2e/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:10.425586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d9a0f4-d80e-44fd-9ea6-886917959e77 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db8d12b797ca4982ad0ebbc95384040a, user_id=2f37f72c6cdf48ad9415b6b9297b9f2e, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:10.429331 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d9a0f4-d80e-44fd-9ea6-886917959e77 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:10.477562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1137/2278] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:11:10 2022] PUT /identity/v3/projects/db8d12b797ca4982ad0ebbc95384040a/users/2f37f72c6cdf48ad9415b6b9297b9f2e/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:10.481705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfb1b9c-71fc-4a6e-8ccf-2577fc16a34c admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:10.492034 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfb1b9c-71fc-4a6e-8ccf-2577fc16a34c admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:10.512536 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:10.558660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a58c1504-ac79-43eb-b05c-5f988cc3ebe1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:10.560220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58c1504-ac79-43eb-b05c-5f988cc3ebe1 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:10.560220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58c1504-ac79-43eb-b05c-5f988cc3ebe1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:10.560220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a58c1504-ac79-43eb-b05c-5f988cc3ebe1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:10.561103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58c1504-ac79-43eb-b05c-5f988cc3ebe1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:10.562991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a58c1504-ac79-43eb-b05c-5f988cc3ebe1 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:10.573999 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1142/2279] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:10 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:11:10.578951 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d9a0f4-d80e-44fd-9ea6-886917959e77 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:10.584350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:10.589619 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d9a0f4-d80e-44fd-9ea6-886917959e77 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:10.594889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42b9359-641e-48a5-8925-9cb515eff213 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:10.595544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42b9359-641e-48a5-8925-9cb515eff213 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:10.595845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42b9359-641e-48a5-8925-9cb515eff213 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:10.596164 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42b9359-641e-48a5-8925-9cb515eff213 admin admin] PATH_INFO: `/v3/projects/db8d12b797ca4982ad0ebbc95384040a/users/2f37f72c6cdf48ad9415b6b9297b9f2e/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:10.598017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42b9359-641e-48a5-8925-9cb515eff213 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db8d12b797ca4982ad0ebbc95384040a, user_id=2f37f72c6cdf48ad9415b6b9297b9f2e, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:10.600711 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42b9359-641e-48a5-8925-9cb515eff213 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:10.614024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1138/2280] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:11:10 2022] PUT /identity/v3/projects/db8d12b797ca4982ad0ebbc95384040a/users/2f37f72c6cdf48ad9415b6b9297b9f2e/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:10.624714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60ea72e-e1ce-46e2-b3b8-964e7eca0cc2 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:10.625127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60ea72e-e1ce-46e2-b3b8-964e7eca0cc2 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:10.625432 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60ea72e-e1ce-46e2-b3b8-964e7eca0cc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:10.655931 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-f60ea72e-e1ce-46e2-b3b8-964e7eca0cc2 None None] MFA Rules not processed for user `2f37f72c6cdf48ad9415b6b9297b9f2e`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:10.657209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-f60ea72e-e1ce-46e2-b3b8-964e7eca0cc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:10.726346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1143/2281] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:10 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:10.740207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42b9359-641e-48a5-8925-9cb515eff213 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:10.741952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f42b9359-641e-48a5-8925-9cb515eff213 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:10.799946 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42b9359-641e-48a5-8925-9cb515eff213 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:10.803556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd24f45-680c-478c-b733-174eec70f39a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:10.804163 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd24f45-680c-478c-b733-174eec70f39a service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:10.804444 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd24f45-680c-478c-b733-174eec70f39a service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:10.804716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd24f45-680c-478c-b733-174eec70f39a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:10.816273 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd24f45-680c-478c-b733-174eec70f39a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:10.817743 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd24f45-680c-478c-b733-174eec70f39a service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:10.828095 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1139/2282] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:11:10 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:11:11.531287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cce652-5fd4-40d2-86e5-c97336dba1f6 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:11.531792 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cce652-5fd4-40d2-86e5-c97336dba1f6 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:11.531924 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cce652-5fd4-40d2-86e5-c97336dba1f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:11.551382 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-26cce652-5fd4-40d2-86e5-c97336dba1f6 None None] MFA Rules not processed for user `f5e4723258c64ce5885154dded2d3d43`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:11.552591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-26cce652-5fd4-40d2-86e5-c97336dba1f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:11.608640 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1144/2283] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:11 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:11.864530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd24f45-680c-478c-b733-174eec70f39a service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:11.874926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd24f45-680c-478c-b733-174eec70f39a service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:11.879586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5871241-615f-4942-95b8-eb8569e1ea77 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f37f72c6cdf48ad9415b6b9297b9f2e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'db8d12b797ca4982ad0ebbc95384040a', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:11.880187 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5871241-615f-4942-95b8-eb8569e1ea77 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:11.881781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5871241-615f-4942-95b8-eb8569e1ea77 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:11.881781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5871241-615f-4942-95b8-eb8569e1ea77 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:11.881781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5871241-615f-4942-95b8-eb8569e1ea77 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:11:11.881781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5871241-615f-4942-95b8-eb8569e1ea77 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:11.883496 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5871241-615f-4942-95b8-eb8569e1ea77 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:11.898560 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1140/2284] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:11:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:11:11.899739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cce652-5fd4-40d2-86e5-c97336dba1f6 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:11.911190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cce652-5fd4-40d2-86e5-c97336dba1f6 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:11.915718 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95d878e-f748-46dc-9e35-c9429b7e1c68 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f37f72c6cdf48ad9415b6b9297b9f2e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'db8d12b797ca4982ad0ebbc95384040a', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:11.916320 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95d878e-f748-46dc-9e35-c9429b7e1c68 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:11.916600 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95d878e-f748-46dc-9e35-c9429b7e1c68 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:11.916870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95d878e-f748-46dc-9e35-c9429b7e1c68 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:11.917550 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95d878e-f748-46dc-9e35-c9429b7e1c68 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:11.918559 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5871241-615f-4942-95b8-eb8569e1ea77 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:11.919456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95d878e-f748-46dc-9e35-c9429b7e1c68 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:11.920843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e5871241-615f-4942-95b8-eb8569e1ea77 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:11.980478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5871241-615f-4942-95b8-eb8569e1ea77 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:11.985105 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53c2c6c-88d6-48b0-aa4a-4704ddfd67e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:11.985702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53c2c6c-88d6-48b0-aa4a-4704ddfd67e6 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:11.985980 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53c2c6c-88d6-48b0-aa4a-4704ddfd67e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:11.986251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53c2c6c-88d6-48b0-aa4a-4704ddfd67e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:11.988211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d53c2c6c-88d6-48b0-aa4a-4704ddfd67e6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:11.998340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53c2c6c-88d6-48b0-aa4a-4704ddfd67e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:11.999744 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53c2c6c-88d6-48b0-aa4a-4704ddfd67e6 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:12.007715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1141/2285] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:11:11 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:11:12.012711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1145/2286] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:11:11 2022] POST /identity/v3/projects => generated 355 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:11:12.013975 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53c2c6c-88d6-48b0-aa4a-4704ddfd67e6 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:12.024946 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53c2c6c-88d6-48b0-aa4a-4704ddfd67e6 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:12.089149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238cf3e6-1568-4488-b884-f7135b9b202c tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f37f72c6cdf48ad9415b6b9297b9f2e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'db8d12b797ca4982ad0ebbc95384040a', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:12.089953 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238cf3e6-1568-4488-b884-f7135b9b202c tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:12.090260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238cf3e6-1568-4488-b884-f7135b9b202c tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:12.090581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238cf3e6-1568-4488-b884-f7135b9b202c tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:12.091406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238cf3e6-1568-4488-b884-f7135b9b202c tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:11:12.092378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238cf3e6-1568-4488-b884-f7135b9b202c tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:12.094381 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238cf3e6-1568-4488-b884-f7135b9b202c tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:12.111375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1142/2287] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:11:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 01:11:12.153263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95d878e-f748-46dc-9e35-c9429b7e1c68 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:12.164176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95d878e-f748-46dc-9e35-c9429b7e1c68 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:12.168736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6fbc0e-fb0f-465d-8425-f241e70ed491 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f37f72c6cdf48ad9415b6b9297b9f2e', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'db8d12b797ca4982ad0ebbc95384040a', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:12.169395 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6fbc0e-fb0f-465d-8425-f241e70ed491 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:12.169681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6fbc0e-fb0f-465d-8425-f241e70ed491 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:12.170008 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6fbc0e-fb0f-465d-8425-f241e70ed491 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] PATH_INFO: `/v3/projects/cc4d7d6e143b44229b5696ea17ce72a2` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:12.171292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6fbc0e-fb0f-465d-8425-f241e70ed491 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] RBAC: Authorizing `identity:delete_project(project_id=cc4d7d6e143b44229b5696ea17ce72a2)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:12.173313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6fbc0e-fb0f-465d-8425-f241e70ed491 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:12.199701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:12.296125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cc6fbc0e-fb0f-465d-8425-f241e70ed491 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cc4d7d6e143b44229b5696ea17ce72a2', 'request_id': 'req-cc6fbc0e-fb0f-465d-8425-f241e70ed491'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:12.301638 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cc6fbc0e-fb0f-465d-8425-f241e70ed491 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cc4d7d6e143b44229b5696ea17ce72a2', 'request_id': 'req-cc6fbc0e-fb0f-465d-8425-f241e70ed491'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:12.307993 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cc6fbc0e-fb0f-465d-8425-f241e70ed491 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cc4d7d6e143b44229b5696ea17ce72a2', 'request_id': 'req-cc6fbc0e-fb0f-465d-8425-f241e70ed491'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:12.392279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cc6fbc0e-fb0f-465d-8425-f241e70ed491 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] The token cache is being invalidate because project cc4d7d6e143b44229b5696ea17ce72a2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:11:12.392690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cc6fbc0e-fb0f-465d-8425-f241e70ed491 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:12.393295 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cc6fbc0e-fb0f-465d-8425-f241e70ed491 tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:12.397843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1146/2288] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:11:12 2022] DELETE /identity/v3/projects/cc4d7d6e143b44229b5696ea17ce72a2 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:13.018178 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238cf3e6-1568-4488-b884-f7135b9b202c tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:13.019940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-238cf3e6-1568-4488-b884-f7135b9b202c tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:13.038684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:13.086456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238cf3e6-1568-4488-b884-f7135b9b202c tempest-VolumeQuotaClassesTest-1961945258 tempest-VolumeQuotaClassesTest-1961945258-project-admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:13.144174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9119899-717f-4498-9439-f1d04edce071 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:13.144839 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9119899-717f-4498-9439-f1d04edce071 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:13.145150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9119899-717f-4498-9439-f1d04edce071 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:13.145428 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9119899-717f-4498-9439-f1d04edce071 admin admin] PATH_INFO: `/v3/users/97ec75d3f9cb4396a7e9ec74ff5554fb` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:13.146554 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9119899-717f-4498-9439-f1d04edce071 admin admin] RBAC: Authorizing `identity:delete_user(user_id=97ec75d3f9cb4396a7e9ec74ff5554fb)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:13.148469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9119899-717f-4498-9439-f1d04edce071 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:13.381266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d9119899-717f-4498-9439-f1d04edce071 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '97ec75d3f9cb4396a7e9ec74ff5554fb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:13.381921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d9119899-717f-4498-9439-f1d04edce071 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '97ec75d3f9cb4396a7e9ec74ff5554fb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:13.471060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d9119899-717f-4498-9439-f1d04edce071 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '97ec75d3f9cb4396a7e9ec74ff5554fb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:13.483863 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d9119899-717f-4498-9439-f1d04edce071 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '97ec75d3f9cb4396a7e9ec74ff5554fb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:13.484586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d9119899-717f-4498-9439-f1d04edce071 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '97ec75d3f9cb4396a7e9ec74ff5554fb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:13.497649 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1143/2289] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:11:13 2022] DELETE /identity/v3/users/97ec75d3f9cb4396a7e9ec74ff5554fb => generated 0 bytes in 480 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:13.502605 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2021beb7-17c6-4230-93f4-22feed4fd892 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:13.504542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2021beb7-17c6-4230-93f4-22feed4fd892 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:13.504542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2021beb7-17c6-4230-93f4-22feed4fd892 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:13.524501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-2021beb7-17c6-4230-93f4-22feed4fd892 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:13.525773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-2021beb7-17c6-4230-93f4-22feed4fd892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:13.603867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1147/2290] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:13.618951 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9119899-717f-4498-9439-f1d04edce071 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:13.621651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d9119899-717f-4498-9439-f1d04edce071 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:13.690746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:13.702723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9119899-717f-4498-9439-f1d04edce071 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:13.707829 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4e7937-f9df-44d2-8e74-b628bc14f885 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:13.708446 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4e7937-f9df-44d2-8e74-b628bc14f885 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:13.708760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4e7937-f9df-44d2-8e74-b628bc14f885 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:13.709033 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4e7937-f9df-44d2-8e74-b628bc14f885 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:13.720037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4e7937-f9df-44d2-8e74-b628bc14f885 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:13.721591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4e7937-f9df-44d2-8e74-b628bc14f885 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:13.733601 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1144/2291] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:11:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:11:14.188459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2021beb7-17c6-4230-93f4-22feed4fd892 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:14.191598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-2021beb7-17c6-4230-93f4-22feed4fd892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:14.208565 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2021beb7-17c6-4230-93f4-22feed4fd892 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:14.213287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5cf3ac-c69c-478e-b507-81c6c179ad37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:14.214169 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5cf3ac-c69c-478e-b507-81c6c179ad37 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:14.214523 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5cf3ac-c69c-478e-b507-81c6c179ad37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:14.214882 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5cf3ac-c69c-478e-b507-81c6c179ad37 admin admin] PATH_INFO: `/v3/projects/116334f5d153436bae64e2cb17629671` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:14.216269 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5cf3ac-c69c-478e-b507-81c6c179ad37 admin admin] RBAC: Authorizing `identity:delete_project(project_id=116334f5d153436bae64e2cb17629671)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:14.219061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5cf3ac-c69c-478e-b507-81c6c179ad37 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:14.336856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-5c5cf3ac-c69c-478e-b507-81c6c179ad37 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '116334f5d153436bae64e2cb17629671', 'request_id': 'req-5c5cf3ac-c69c-478e-b507-81c6c179ad37'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:14.344803 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-5c5cf3ac-c69c-478e-b507-81c6c179ad37 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '116334f5d153436bae64e2cb17629671', 'request_id': 'req-5c5cf3ac-c69c-478e-b507-81c6c179ad37'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:14.352382 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-5c5cf3ac-c69c-478e-b507-81c6c179ad37 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '116334f5d153436bae64e2cb17629671', 'request_id': 'req-5c5cf3ac-c69c-478e-b507-81c6c179ad37'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:14.419873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-5c5cf3ac-c69c-478e-b507-81c6c179ad37 admin admin] The token cache is being invalidate because project 116334f5d153436bae64e2cb17629671 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:11:14.420533 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-5c5cf3ac-c69c-478e-b507-81c6c179ad37 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:14.421447 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-5c5cf3ac-c69c-478e-b507-81c6c179ad37 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:14.423127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1148/2292] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:11:14 2022] DELETE /identity/v3/projects/116334f5d153436bae64e2cb17629671 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:14.427322 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4e7937-f9df-44d2-8e74-b628bc14f885 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:14.440447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4e7937-f9df-44d2-8e74-b628bc14f885 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:14.502538 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43021f43-e29b-4097-a650-c8d05e6b6aa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:14.503204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43021f43-e29b-4097-a650-c8d05e6b6aa2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:14.503488 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43021f43-e29b-4097-a650-c8d05e6b6aa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:14.503777 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43021f43-e29b-4097-a650-c8d05e6b6aa2 admin admin] PATH_INFO: `/v3/users/2f37f72c6cdf48ad9415b6b9297b9f2e` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:14.504905 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43021f43-e29b-4097-a650-c8d05e6b6aa2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f37f72c6cdf48ad9415b6b9297b9f2e)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:14.506852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43021f43-e29b-4097-a650-c8d05e6b6aa2 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:14.728389 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-43021f43-e29b-4097-a650-c8d05e6b6aa2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f37f72c6cdf48ad9415b6b9297b9f2e'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:14.735665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-43021f43-e29b-4097-a650-c8d05e6b6aa2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f37f72c6cdf48ad9415b6b9297b9f2e'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:14.803110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-43021f43-e29b-4097-a650-c8d05e6b6aa2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f37f72c6cdf48ad9415b6b9297b9f2e'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:14.817465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-43021f43-e29b-4097-a650-c8d05e6b6aa2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f37f72c6cdf48ad9415b6b9297b9f2e'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:14.818156 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-43021f43-e29b-4097-a650-c8d05e6b6aa2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f37f72c6cdf48ad9415b6b9297b9f2e'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:14.831015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1145/2293] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:11:14 2022] DELETE /identity/v3/users/2f37f72c6cdf48ad9415b6b9297b9f2e => generated 0 bytes in 404 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:15.352833 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5cf3ac-c69c-478e-b507-81c6c179ad37 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:15.354437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-5c5cf3ac-c69c-478e-b507-81c6c179ad37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:15.368590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5cf3ac-c69c-478e-b507-81c6c179ad37 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:15.407536 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:15.435234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92df525d-0d18-4511-8dc7-d2fdbad1d5c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:15.436848 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92df525d-0d18-4511-8dc7-d2fdbad1d5c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:15.437315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92df525d-0d18-4511-8dc7-d2fdbad1d5c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:15.437594 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92df525d-0d18-4511-8dc7-d2fdbad1d5c5 admin admin] PATH_INFO: `/v3/projects/db8d12b797ca4982ad0ebbc95384040a` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:15.438837 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92df525d-0d18-4511-8dc7-d2fdbad1d5c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=db8d12b797ca4982ad0ebbc95384040a)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:15.440942 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92df525d-0d18-4511-8dc7-d2fdbad1d5c5 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:15.523962 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-92df525d-0d18-4511-8dc7-d2fdbad1d5c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db8d12b797ca4982ad0ebbc95384040a', 'request_id': 'req-92df525d-0d18-4511-8dc7-d2fdbad1d5c5'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:15.530007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-92df525d-0d18-4511-8dc7-d2fdbad1d5c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db8d12b797ca4982ad0ebbc95384040a', 'request_id': 'req-92df525d-0d18-4511-8dc7-d2fdbad1d5c5'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:15.536363 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-92df525d-0d18-4511-8dc7-d2fdbad1d5c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db8d12b797ca4982ad0ebbc95384040a', 'request_id': 'req-92df525d-0d18-4511-8dc7-d2fdbad1d5c5'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:15.631278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-92df525d-0d18-4511-8dc7-d2fdbad1d5c5 admin admin] The token cache is being invalidate because project db8d12b797ca4982ad0ebbc95384040a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:11:15.631732 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-92df525d-0d18-4511-8dc7-d2fdbad1d5c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:15.632537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-92df525d-0d18-4511-8dc7-d2fdbad1d5c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:15.634215 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1149/2294] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:11:15 2022] DELETE /identity/v3/projects/db8d12b797ca4982ad0ebbc95384040a => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:15.640181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e403bb-9f9b-4a2c-8291-0f83f10a10cc None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:15.640695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e403bb-9f9b-4a2c-8291-0f83f10a10cc None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:15.640824 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e403bb-9f9b-4a2c-8291-0f83f10a10cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:15.661335 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-75e403bb-9f9b-4a2c-8291-0f83f10a10cc None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:15.662719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-75e403bb-9f9b-4a2c-8291-0f83f10a10cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:15.729152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1146/2295] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:15.736907 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d10df48-ad8a-4d7c-b0c8-fe28084ec798 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:15.737228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d10df48-ad8a-4d7c-b0c8-fe28084ec798 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:15.737526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d10df48-ad8a-4d7c-b0c8-fe28084ec798 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:15.762583 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-4d10df48-ad8a-4d7c-b0c8-fe28084ec798 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:15.766233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4d10df48-ad8a-4d7c-b0c8-fe28084ec798 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:15.774743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1150/2296] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:15.778904 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e403bb-9f9b-4a2c-8291-0f83f10a10cc None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:15.785964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:15.790008 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e403bb-9f9b-4a2c-8291-0f83f10a10cc None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:15.795495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77f5513-8a6a-4704-8856-7a39d6ece54d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:15.796179 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77f5513-8a6a-4704-8856-7a39d6ece54d admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:15.796531 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77f5513-8a6a-4704-8856-7a39d6ece54d admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:15.796873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77f5513-8a6a-4704-8856-7a39d6ece54d admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:15.797653 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77f5513-8a6a-4704-8856-7a39d6ece54d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:11:15.798014 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77f5513-8a6a-4704-8856-7a39d6ece54d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:15.799700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77f5513-8a6a-4704-8856-7a39d6ece54d admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:15.813188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1147/2297] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:11:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:11:15.817916 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d10df48-ad8a-4d7c-b0c8-fe28084ec798 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:15.846235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d10df48-ad8a-4d7c-b0c8-fe28084ec798 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:15.846235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c860ff-bccd-4e05-be94-a5f978c1b3e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:15.846235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c860ff-bccd-4e05-be94-a5f978c1b3e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:15.846235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c860ff-bccd-4e05-be94-a5f978c1b3e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:15.846606 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c860ff-bccd-4e05-be94-a5f978c1b3e6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:15.846606 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c860ff-bccd-4e05-be94-a5f978c1b3e6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:15.846606 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c860ff-bccd-4e05-be94-a5f978c1b3e6 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:15.945347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1151/2298] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:11:15 2022] POST /identity/v3/projects => generated 365 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:11:15.946805 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77f5513-8a6a-4704-8856-7a39d6ece54d admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:15.969044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77f5513-8a6a-4704-8856-7a39d6ece54d admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:16.082699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38645862-0c42-4c42-ad5d-8638111aef08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:16.083633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38645862-0c42-4c42-ad5d-8638111aef08 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:16.084128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38645862-0c42-4c42-ad5d-8638111aef08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:16.084373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38645862-0c42-4c42-ad5d-8638111aef08 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:16.085223 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38645862-0c42-4c42-ad5d-8638111aef08 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:16.087438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38645862-0c42-4c42-ad5d-8638111aef08 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:16.212983 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1148/2299] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:11:15 2022] POST /identity/v3/users => generated 283 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:11:16.214092 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c860ff-bccd-4e05-be94-a5f978c1b3e6 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:16.227477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c860ff-bccd-4e05-be94-a5f978c1b3e6 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:16.233471 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e388b81-b863-41cd-a7d9-6478f544aa29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:16.234568 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e388b81-b863-41cd-a7d9-6478f544aa29 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:16.235084 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e388b81-b863-41cd-a7d9-6478f544aa29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:16.235500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e388b81-b863-41cd-a7d9-6478f544aa29 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:16.236153 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e388b81-b863-41cd-a7d9-6478f544aa29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:16.238434 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e388b81-b863-41cd-a7d9-6478f544aa29 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:16.254901 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1152/2300] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:16 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:11:16.260183 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38645862-0c42-4c42-ad5d-8638111aef08 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:16.272833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38645862-0c42-4c42-ad5d-8638111aef08 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:16.280146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26995027-d17c-4f2f-b61d-e35091f2d331 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:16.281285 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26995027-d17c-4f2f-b61d-e35091f2d331 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:16.281592 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26995027-d17c-4f2f-b61d-e35091f2d331 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:16.281865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26995027-d17c-4f2f-b61d-e35091f2d331 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:16.282525 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26995027-d17c-4f2f-b61d-e35091f2d331 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:16.284344 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26995027-d17c-4f2f-b61d-e35091f2d331 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:16.297452 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1149/2301] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:16 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:11:16.302132 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e388b81-b863-41cd-a7d9-6478f544aa29 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:16.314842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e388b81-b863-41cd-a7d9-6478f544aa29 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:16.320174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9361ba10-91d7-4ed3-b498-ed3a8055678d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:16.320840 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9361ba10-91d7-4ed3-b498-ed3a8055678d admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:16.321147 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9361ba10-91d7-4ed3-b498-ed3a8055678d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:16.321425 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9361ba10-91d7-4ed3-b498-ed3a8055678d admin admin] PATH_INFO: `/v3/projects/51bc35cf23b04168b0ce60cec276724f/users/6d4c0b22638a4fea86ae5eedf0b4e363/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:16.338577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9361ba10-91d7-4ed3-b498-ed3a8055678d admin admin] RBAC: Authorizing `identity:create_grant(project_id=51bc35cf23b04168b0ce60cec276724f, user_id=6d4c0b22638a4fea86ae5eedf0b4e363, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:16.341281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9361ba10-91d7-4ed3-b498-ed3a8055678d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:16.398807 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1153/2302] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:11:16 2022] PUT /identity/v3/projects/51bc35cf23b04168b0ce60cec276724f/users/6d4c0b22638a4fea86ae5eedf0b4e363/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:11:16.410850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bd3c79-85af-4d8f-a84a-6dc93012bf3e None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:16.411135 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bd3c79-85af-4d8f-a84a-6dc93012bf3e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:16.411399 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bd3c79-85af-4d8f-a84a-6dc93012bf3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:16.444966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-d1bd3c79-85af-4d8f-a84a-6dc93012bf3e None None] MFA Rules not processed for user `6d4c0b22638a4fea86ae5eedf0b4e363`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:16.446254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d1bd3c79-85af-4d8f-a84a-6dc93012bf3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:16.528849 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1150/2303] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:16 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:16.533495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9361ba10-91d7-4ed3-b498-ed3a8055678d admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:16.604281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9361ba10-91d7-4ed3-b498-ed3a8055678d admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:16.609541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb4caa0-853c-41f8-b137-25ed0405154e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:16.612026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb4caa0-853c-41f8-b137-25ed0405154e admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:16.612026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb4caa0-853c-41f8-b137-25ed0405154e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:16.612026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb4caa0-853c-41f8-b137-25ed0405154e admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:16.612026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb4caa0-853c-41f8-b137-25ed0405154e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:16.614882 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb4caa0-853c-41f8-b137-25ed0405154e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:16.623994 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:16.762880 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1154/2304] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:11:16 2022] POST /identity/v3/projects => generated 367 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:11:16.767693 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bd3c79-85af-4d8f-a84a-6dc93012bf3e None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:16.778737 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bd3c79-85af-4d8f-a84a-6dc93012bf3e None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:16.866813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4bf712-383c-48a0-a7d0-8f5c34304da5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:16.867469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4bf712-383c-48a0-a7d0-8f5c34304da5 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:16.867749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4bf712-383c-48a0-a7d0-8f5c34304da5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:16.868023 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4bf712-383c-48a0-a7d0-8f5c34304da5 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:16.868662 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4bf712-383c-48a0-a7d0-8f5c34304da5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:16.870720 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4bf712-383c-48a0-a7d0-8f5c34304da5 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:16.879122 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:16.975154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1151/2305] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:11:16 2022] POST /identity/v3/users => generated 290 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:11:16.979427 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb4caa0-853c-41f8-b137-25ed0405154e admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:16.991277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb4caa0-853c-41f8-b137-25ed0405154e admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:16.996004 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e799f79b-7b34-40fc-b90d-c8ae5cf1065e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:16.996615 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e799f79b-7b34-40fc-b90d-c8ae5cf1065e admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:16.996892 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e799f79b-7b34-40fc-b90d-c8ae5cf1065e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:16.997208 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e799f79b-7b34-40fc-b90d-c8ae5cf1065e admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:16.997861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e799f79b-7b34-40fc-b90d-c8ae5cf1065e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:16.999523 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e799f79b-7b34-40fc-b90d-c8ae5cf1065e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:17.010098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1155/2306] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:16 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:11:17.030881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4bf712-383c-48a0-a7d0-8f5c34304da5 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:17.042325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4bf712-383c-48a0-a7d0-8f5c34304da5 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:17.048060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe610de-eda3-4e48-b74b-12555e96501b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:17.048952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe610de-eda3-4e48-b74b-12555e96501b admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:17.049439 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe610de-eda3-4e48-b74b-12555e96501b admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:17.049718 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe610de-eda3-4e48-b74b-12555e96501b admin admin] PATH_INFO: `/v3/projects/85eb051f40134bc287ff73cd0f2ffb00/users/e8f34ac9f81a46ce8c9a3f03f8dbb584/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:17.069574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe610de-eda3-4e48-b74b-12555e96501b admin admin] RBAC: Authorizing `identity:create_grant(project_id=85eb051f40134bc287ff73cd0f2ffb00, user_id=e8f34ac9f81a46ce8c9a3f03f8dbb584, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:17.072341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe610de-eda3-4e48-b74b-12555e96501b admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:17.169110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1152/2307] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:11:17 2022] PUT /identity/v3/projects/85eb051f40134bc287ff73cd0f2ffb00/users/e8f34ac9f81a46ce8c9a3f03f8dbb584/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:17.173732 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e799f79b-7b34-40fc-b90d-c8ae5cf1065e admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:17.208236 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e799f79b-7b34-40fc-b90d-c8ae5cf1065e admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:17.261843 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f3b084-3a42-4adc-ac3f-43f546b2ccac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:17.262963 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f3b084-3a42-4adc-ac3f-43f546b2ccac admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:17.263315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f3b084-3a42-4adc-ac3f-43f546b2ccac admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:17.263683 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f3b084-3a42-4adc-ac3f-43f546b2ccac admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:17.264431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f3b084-3a42-4adc-ac3f-43f546b2ccac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:17.266522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f3b084-3a42-4adc-ac3f-43f546b2ccac admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:17.282951 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1156/2308] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:17 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:11:17.284295 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe610de-eda3-4e48-b74b-12555e96501b admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:17.301016 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe610de-eda3-4e48-b74b-12555e96501b admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:17.306725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05af6ea-1448-4ff9-a0cb-d46826813b09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:17.307695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05af6ea-1448-4ff9-a0cb-d46826813b09 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:17.308100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05af6ea-1448-4ff9-a0cb-d46826813b09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:17.308448 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05af6ea-1448-4ff9-a0cb-d46826813b09 admin admin] PATH_INFO: `/v3/projects/85eb051f40134bc287ff73cd0f2ffb00/users/e8f34ac9f81a46ce8c9a3f03f8dbb584/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:17.311201 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05af6ea-1448-4ff9-a0cb-d46826813b09 admin admin] RBAC: Authorizing `identity:create_grant(project_id=85eb051f40134bc287ff73cd0f2ffb00, user_id=e8f34ac9f81a46ce8c9a3f03f8dbb584, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:17.315078 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05af6ea-1448-4ff9-a0cb-d46826813b09 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:17.331066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1153/2309] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:11:17 2022] PUT /identity/v3/projects/85eb051f40134bc287ff73cd0f2ffb00/users/e8f34ac9f81a46ce8c9a3f03f8dbb584/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:17.343067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d8f940-a627-4188-b092-356f4bb5c6a8 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:17.343549 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d8f940-a627-4188-b092-356f4bb5c6a8 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:17.343721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d8f940-a627-4188-b092-356f4bb5c6a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:17.379314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-e7d8f940-a627-4188-b092-356f4bb5c6a8 None None] MFA Rules not processed for user `e8f34ac9f81a46ce8c9a3f03f8dbb584`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:17.380624 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e7d8f940-a627-4188-b092-356f4bb5c6a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:17.467999 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1157/2310] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:17 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:17.479883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05af6ea-1448-4ff9-a0cb-d46826813b09 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:17.482126 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f05af6ea-1448-4ff9-a0cb-d46826813b09 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:17.549206 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05af6ea-1448-4ff9-a0cb-d46826813b09 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:17.552951 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b8e145-5ba9-4489-a29f-837699f5e368 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:17.553601 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b8e145-5ba9-4489-a29f-837699f5e368 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:17.556545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b8e145-5ba9-4489-a29f-837699f5e368 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:17.556545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b8e145-5ba9-4489-a29f-837699f5e368 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:17.563949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b8e145-5ba9-4489-a29f-837699f5e368 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:17.565417 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b8e145-5ba9-4489-a29f-837699f5e368 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:17.574839 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1154/2311] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:11:17 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:11:17.857099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d8f940-a627-4188-b092-356f4bb5c6a8 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:17.867776 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d8f940-a627-4188-b092-356f4bb5c6a8 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:17.871486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c3011e-1c62-4259-9dac-ec105830a07b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:17.875387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c3011e-1c62-4259-9dac-ec105830a07b service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:17.875387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c3011e-1c62-4259-9dac-ec105830a07b service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:17.875387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c3011e-1c62-4259-9dac-ec105830a07b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:17.928007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:17.930406 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c3011e-1c62-4259-9dac-ec105830a07b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:17.931922 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c3011e-1c62-4259-9dac-ec105830a07b service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:17.941720 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1158/2312] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:11:17 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:11:21.919226 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc47ec88-69ee-4d9c-9889-d859761ca083 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:21.919746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc47ec88-69ee-4d9c-9889-d859761ca083 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:21.920424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc47ec88-69ee-4d9c-9889-d859761ca083 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:21.941052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-fc47ec88-69ee-4d9c-9889-d859761ca083 None None] MFA Rules not processed for user `70cef7c271e046a3b49230057cc2d28f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:21.942337 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-fc47ec88-69ee-4d9c-9889-d859761ca083 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:22.039019 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1155/2313] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 4 01:11:21 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:22.224439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c3011e-1c62-4259-9dac-ec105830a07b service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:22.225937 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-22c3011e-1c62-4259-9dac-ec105830a07b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:22.301045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c3011e-1c62-4259-9dac-ec105830a07b service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:22.306288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc72389c-77b0-4f09-a05a-0ca21feb79e8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '123ab6ff796c4e599ce1d6f4095bd898', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:22.314153 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc72389c-77b0-4f09-a05a-0ca21feb79e8 service swift] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:22.314153 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc72389c-77b0-4f09-a05a-0ca21feb79e8 service swift] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:22.314153 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc72389c-77b0-4f09-a05a-0ca21feb79e8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:22.317242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc72389c-77b0-4f09-a05a-0ca21feb79e8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:22.318845 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc72389c-77b0-4f09-a05a-0ca21feb79e8 service swift] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:22.327237 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1159/2314] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 4 01:11:22 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 04 01:11:23.797152 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc47ec88-69ee-4d9c-9889-d859761ca083 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:23.801651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-fc47ec88-69ee-4d9c-9889-d859761ca083 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:23.877979 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:23.900827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc47ec88-69ee-4d9c-9889-d859761ca083 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:23.909375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f195b2-a4fa-48c2-910a-59e2b5476748 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:23.910219 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f195b2-a4fa-48c2-910a-59e2b5476748 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:23.910672 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f195b2-a4fa-48c2-910a-59e2b5476748 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:23.911050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f195b2-a4fa-48c2-910a-59e2b5476748 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:23.913147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a7f195b2-a4fa-48c2-910a-59e2b5476748 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:23.931472 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f195b2-a4fa-48c2-910a-59e2b5476748 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:23.936459 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f195b2-a4fa-48c2-910a-59e2b5476748 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:23.957236 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1156/2315] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:11:23 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:11:24.804012 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc72389c-77b0-4f09-a05a-0ca21feb79e8 service swift] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:24.898636 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc72389c-77b0-4f09-a05a-0ca21feb79e8 service swift] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:24.903430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4b2eb5-d863-46cb-ae9d-386784bce6cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:24.904645 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4b2eb5-d863-46cb-ae9d-386784bce6cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:24.904927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4b2eb5-d863-46cb-ae9d-386784bce6cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:24.905241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4b2eb5-d863-46cb-ae9d-386784bce6cf admin admin] PATH_INFO: `/v3/users/6d4c0b22638a4fea86ae5eedf0b4e363` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:24.906458 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4b2eb5-d863-46cb-ae9d-386784bce6cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d4c0b22638a4fea86ae5eedf0b4e363)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:24.908597 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4b2eb5-d863-46cb-ae9d-386784bce6cf admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:25.123930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-9b4b2eb5-d863-46cb-ae9d-386784bce6cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d4c0b22638a4fea86ae5eedf0b4e363'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:25.431761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-9b4b2eb5-d863-46cb-ae9d-386784bce6cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d4c0b22638a4fea86ae5eedf0b4e363'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:25.448152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-9b4b2eb5-d863-46cb-ae9d-386784bce6cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d4c0b22638a4fea86ae5eedf0b4e363'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:25.449503 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-9b4b2eb5-d863-46cb-ae9d-386784bce6cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d4c0b22638a4fea86ae5eedf0b4e363'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:25.457537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-9b4b2eb5-d863-46cb-ae9d-386784bce6cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d4c0b22638a4fea86ae5eedf0b4e363'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:25.468821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1160/2316] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:11:24 2022] DELETE /identity/v3/users/6d4c0b22638a4fea86ae5eedf0b4e363 => generated 0 bytes in 665 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:25.473577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f552dd-e3a6-4156-81b5-fee05f1edc1e None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:25.474073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f552dd-e3a6-4156-81b5-fee05f1edc1e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:25.474304 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f552dd-e3a6-4156-81b5-fee05f1edc1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:25.497501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-a7f552dd-e3a6-4156-81b5-fee05f1edc1e None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:25.498836 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a7f552dd-e3a6-4156-81b5-fee05f1edc1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:25.591020 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1157/2317] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:25.601564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4b2eb5-d863-46cb-ae9d-386784bce6cf admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:25.603002 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9b4b2eb5-d863-46cb-ae9d-386784bce6cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:25.619107 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:25.685636 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4b2eb5-d863-46cb-ae9d-386784bce6cf admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:25.700722 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6274b21b-b064-4479-994a-12df85a3d6dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:25.701712 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6274b21b-b064-4479-994a-12df85a3d6dd service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:25.702180 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6274b21b-b064-4479-994a-12df85a3d6dd service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:25.702654 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6274b21b-b064-4479-994a-12df85a3d6dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:25.714824 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6274b21b-b064-4479-994a-12df85a3d6dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:25.714824 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6274b21b-b064-4479-994a-12df85a3d6dd service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:25.737545 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1161/2318] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:11:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:11:26.366550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f552dd-e3a6-4156-81b5-fee05f1edc1e None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:26.368047 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a7f552dd-e3a6-4156-81b5-fee05f1edc1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:26.382302 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f552dd-e3a6-4156-81b5-fee05f1edc1e None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:26.399025 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59738057-4770-4d30-b04e-5e3006832b1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:26.400179 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59738057-4770-4d30-b04e-5e3006832b1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:26.401431 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59738057-4770-4d30-b04e-5e3006832b1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:26.401818 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59738057-4770-4d30-b04e-5e3006832b1b admin admin] PATH_INFO: `/v3/projects/51bc35cf23b04168b0ce60cec276724f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:26.403111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59738057-4770-4d30-b04e-5e3006832b1b admin admin] RBAC: Authorizing `identity:delete_project(project_id=51bc35cf23b04168b0ce60cec276724f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:26.406087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59738057-4770-4d30-b04e-5e3006832b1b admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:26.509800 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-59738057-4770-4d30-b04e-5e3006832b1b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '51bc35cf23b04168b0ce60cec276724f', 'request_id': 'req-59738057-4770-4d30-b04e-5e3006832b1b'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:26.517224 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-59738057-4770-4d30-b04e-5e3006832b1b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '51bc35cf23b04168b0ce60cec276724f', 'request_id': 'req-59738057-4770-4d30-b04e-5e3006832b1b'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:26.588426 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-59738057-4770-4d30-b04e-5e3006832b1b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '51bc35cf23b04168b0ce60cec276724f', 'request_id': 'req-59738057-4770-4d30-b04e-5e3006832b1b'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:26.613923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-59738057-4770-4d30-b04e-5e3006832b1b admin admin] The token cache is being invalidate because project 51bc35cf23b04168b0ce60cec276724f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:11:26.618725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-59738057-4770-4d30-b04e-5e3006832b1b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:26.618725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-59738057-4770-4d30-b04e-5e3006832b1b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:26.621801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6274b21b-b064-4479-994a-12df85a3d6dd service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:26.635213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1158/2319] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:11:26 2022] DELETE /identity/v3/projects/51bc35cf23b04168b0ce60cec276724f => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:26.640721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6274b21b-b064-4479-994a-12df85a3d6dd service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:26.776829 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d14788e-41ee-4d10-8878-125799849abd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:26.777980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d14788e-41ee-4d10-8878-125799849abd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:26.780070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d14788e-41ee-4d10-8878-125799849abd admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:26.780548 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d14788e-41ee-4d10-8878-125799849abd admin admin] PATH_INFO: `/v3/users/e8f34ac9f81a46ce8c9a3f03f8dbb584` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:26.782653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d14788e-41ee-4d10-8878-125799849abd admin admin] RBAC: Authorizing `identity:delete_user(user_id=e8f34ac9f81a46ce8c9a3f03f8dbb584)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:26.789428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d14788e-41ee-4d10-8878-125799849abd admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:26.977913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0d14788e-41ee-4d10-8878-125799849abd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e8f34ac9f81a46ce8c9a3f03f8dbb584'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:27.117416 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0d14788e-41ee-4d10-8878-125799849abd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e8f34ac9f81a46ce8c9a3f03f8dbb584'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:27.139470 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0d14788e-41ee-4d10-8878-125799849abd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e8f34ac9f81a46ce8c9a3f03f8dbb584'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:27.143385 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0d14788e-41ee-4d10-8878-125799849abd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e8f34ac9f81a46ce8c9a3f03f8dbb584'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:27.156796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0d14788e-41ee-4d10-8878-125799849abd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e8f34ac9f81a46ce8c9a3f03f8dbb584'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:27.177381 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1162/2320] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:11:26 2022] DELETE /identity/v3/users/e8f34ac9f81a46ce8c9a3f03f8dbb584 => generated 0 bytes in 553 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:27.740333 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59738057-4770-4d30-b04e-5e3006832b1b admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:27.743060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-59738057-4770-4d30-b04e-5e3006832b1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:27.759294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59738057-4770-4d30-b04e-5e3006832b1b admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:27.846206 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1abf51-f79c-408c-ba09-6222627252ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:27.846896 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1abf51-f79c-408c-ba09-6222627252ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:27.847190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1abf51-f79c-408c-ba09-6222627252ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:27.847467 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1abf51-f79c-408c-ba09-6222627252ba admin admin] PATH_INFO: `/v3/projects/85eb051f40134bc287ff73cd0f2ffb00` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:27.849309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1abf51-f79c-408c-ba09-6222627252ba admin admin] RBAC: Authorizing `identity:delete_project(project_id=85eb051f40134bc287ff73cd0f2ffb00)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:27.851516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1abf51-f79c-408c-ba09-6222627252ba admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:27.868739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:27.981368 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-be1abf51-f79c-408c-ba09-6222627252ba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '85eb051f40134bc287ff73cd0f2ffb00', 'request_id': 'req-be1abf51-f79c-408c-ba09-6222627252ba'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:27.989042 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-be1abf51-f79c-408c-ba09-6222627252ba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '85eb051f40134bc287ff73cd0f2ffb00', 'request_id': 'req-be1abf51-f79c-408c-ba09-6222627252ba'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:28.065728 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-be1abf51-f79c-408c-ba09-6222627252ba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '85eb051f40134bc287ff73cd0f2ffb00', 'request_id': 'req-be1abf51-f79c-408c-ba09-6222627252ba'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:28.083686 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-be1abf51-f79c-408c-ba09-6222627252ba admin admin] The token cache is being invalidate because project 85eb051f40134bc287ff73cd0f2ffb00 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:11:28.084034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-be1abf51-f79c-408c-ba09-6222627252ba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:28.085043 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-be1abf51-f79c-408c-ba09-6222627252ba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:28.086917 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1159/2321] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:11:27 2022] DELETE /identity/v3/projects/85eb051f40134bc287ff73cd0f2ffb00 => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:28.091782 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cbbb94-38c4-462d-8ed1-fa6680b1dccb None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:28.092302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cbbb94-38c4-462d-8ed1-fa6680b1dccb None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:28.092725 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cbbb94-38c4-462d-8ed1-fa6680b1dccb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:28.125528 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-a6cbbb94-38c4-462d-8ed1-fa6680b1dccb None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:28.126709 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a6cbbb94-38c4-462d-8ed1-fa6680b1dccb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:28.170473 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:28.225347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1163/2322] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:11:28.234254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98dbde87-a7e6-4e3f-98df-4887f583cbc8 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:28.234819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98dbde87-a7e6-4e3f-98df-4887f583cbc8 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:28.235460 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98dbde87-a7e6-4e3f-98df-4887f583cbc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:28.284532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-98dbde87-a7e6-4e3f-98df-4887f583cbc8 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:28.286828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-98dbde87-a7e6-4e3f-98df-4887f583cbc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:28.301579 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1160/2323] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:28.307012 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cbbb94-38c4-462d-8ed1-fa6680b1dccb None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:28.336691 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cbbb94-38c4-462d-8ed1-fa6680b1dccb None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:28.350794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7f2521-9dae-48e4-8515-e2beb96cde6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:28.351419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7f2521-9dae-48e4-8515-e2beb96cde6a admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:28.355322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7f2521-9dae-48e4-8515-e2beb96cde6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:28.355322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7f2521-9dae-48e4-8515-e2beb96cde6a admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:28.355322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7f2521-9dae-48e4-8515-e2beb96cde6a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:11:28.355322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7f2521-9dae-48e4-8515-e2beb96cde6a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:28.355322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7f2521-9dae-48e4-8515-e2beb96cde6a admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:28.378169 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1164/2324] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:11:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:11:28.383654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98dbde87-a7e6-4e3f-98df-4887f583cbc8 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:28.400263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98dbde87-a7e6-4e3f-98df-4887f583cbc8 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:28.417673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598e84d6-da1b-4d3b-8711-5410a08aabef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:28.418426 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598e84d6-da1b-4d3b-8711-5410a08aabef admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:28.418803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598e84d6-da1b-4d3b-8711-5410a08aabef admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:28.419345 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598e84d6-da1b-4d3b-8711-5410a08aabef admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:28.421556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598e84d6-da1b-4d3b-8711-5410a08aabef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:28.427109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598e84d6-da1b-4d3b-8711-5410a08aabef admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:28.542248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1161/2325] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:11:28 2022] POST /identity/v3/projects => generated 377 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:11:28.546091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7f2521-9dae-48e4-8515-e2beb96cde6a admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:28.576980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7f2521-9dae-48e4-8515-e2beb96cde6a admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:28.683200 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dcf088-89ad-4b39-849f-71462cd9d607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:28.684013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dcf088-89ad-4b39-849f-71462cd9d607 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:28.684298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dcf088-89ad-4b39-849f-71462cd9d607 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:28.684571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dcf088-89ad-4b39-849f-71462cd9d607 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:28.685728 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3dcf088-89ad-4b39-849f-71462cd9d607 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:28.690325 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3dcf088-89ad-4b39-849f-71462cd9d607 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:28.847896 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1165/2326] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:11:28 2022] POST /identity/v3/users => generated 289 bytes in 304 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:11:28.852368 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598e84d6-da1b-4d3b-8711-5410a08aabef admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:28.864077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598e84d6-da1b-4d3b-8711-5410a08aabef admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:28.870344 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac6e0f4-380d-41de-93a4-9053876f750c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:28.870474 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac6e0f4-380d-41de-93a4-9053876f750c admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:28.870979 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac6e0f4-380d-41de-93a4-9053876f750c admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:28.870979 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac6e0f4-380d-41de-93a4-9053876f750c admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:28.871903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac6e0f4-380d-41de-93a4-9053876f750c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:28.874128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac6e0f4-380d-41de-93a4-9053876f750c admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:28.886654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1162/2327] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:28 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:11:28.891201 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dcf088-89ad-4b39-849f-71462cd9d607 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:28.904716 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dcf088-89ad-4b39-849f-71462cd9d607 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:28.917915 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8007b7ef-b35e-4135-bb47-7fa82153e8db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:28.919157 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8007b7ef-b35e-4135-bb47-7fa82153e8db admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:28.919157 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8007b7ef-b35e-4135-bb47-7fa82153e8db admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:28.919406 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8007b7ef-b35e-4135-bb47-7fa82153e8db admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:28.920482 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8007b7ef-b35e-4135-bb47-7fa82153e8db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:28.922472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8007b7ef-b35e-4135-bb47-7fa82153e8db admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:28.936417 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1166/2328] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:28 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:11:28.944709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac6e0f4-380d-41de-93a4-9053876f750c admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:28.964876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac6e0f4-380d-41de-93a4-9053876f750c admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:28.978370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555ddfdc-d993-46d6-a1fd-ccb2e58b6335 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:28.979793 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555ddfdc-d993-46d6-a1fd-ccb2e58b6335 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:28.980304 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555ddfdc-d993-46d6-a1fd-ccb2e58b6335 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:28.980584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555ddfdc-d993-46d6-a1fd-ccb2e58b6335 admin admin] PATH_INFO: `/v3/projects/57b717c160e64aca8571ae6cbeeee26c/users/538a540a102842a99dfac6ae9264243e/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:29.010207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555ddfdc-d993-46d6-a1fd-ccb2e58b6335 admin admin] RBAC: Authorizing `identity:create_grant(project_id=57b717c160e64aca8571ae6cbeeee26c, user_id=538a540a102842a99dfac6ae9264243e, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:29.014181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555ddfdc-d993-46d6-a1fd-ccb2e58b6335 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:29.125695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1163/2329] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:11:28 2022] PUT /identity/v3/projects/57b717c160e64aca8571ae6cbeeee26c/users/538a540a102842a99dfac6ae9264243e/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:29.139816 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0986a8d2-928b-4eeb-821b-2af24d4a627d None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:29.141764 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0986a8d2-928b-4eeb-821b-2af24d4a627d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:29.141764 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0986a8d2-928b-4eeb-821b-2af24d4a627d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:29.233652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-0986a8d2-928b-4eeb-821b-2af24d4a627d None None] MFA Rules not processed for user `538a540a102842a99dfac6ae9264243e`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:29.241598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0986a8d2-928b-4eeb-821b-2af24d4a627d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:29.329154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1167/2330] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:29 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:29.340133 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555ddfdc-d993-46d6-a1fd-ccb2e58b6335 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:29.342547 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-555ddfdc-d993-46d6-a1fd-ccb2e58b6335 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:29.522207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555ddfdc-d993-46d6-a1fd-ccb2e58b6335 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:29.527035 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c97d95b-2bcb-4af7-a39a-558d3532e4e7 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '69153344fc4b45588bdbcecdfcfd7c3f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:29.527828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c97d95b-2bcb-4af7-a39a-558d3532e4e7 service glance] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:29.528199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c97d95b-2bcb-4af7-a39a-558d3532e4e7 service glance] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:29.528519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c97d95b-2bcb-4af7-a39a-558d3532e4e7 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:29.541911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c97d95b-2bcb-4af7-a39a-558d3532e4e7 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:29.543428 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c97d95b-2bcb-4af7-a39a-558d3532e4e7 service glance] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:29.577727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1164/2331] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:11:29 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:11:39.324146 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=87,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:39.550070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ecb96c-843b-4b98-9eb2-370f59f330a6 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:39.550070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ecb96c-843b-4b98-9eb2-370f59f330a6 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:39.550070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ecb96c-843b-4b98-9eb2-370f59f330a6 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:39.551138 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1168/2332] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 4 01:11:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:11:39.557253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd1b32c-4a48-49f0-af98-c39705ce3966 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:39.557801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd1b32c-4a48-49f0-af98-c39705ce3966 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:39.558072 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=73,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:39.558717 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd1b32c-4a48-49f0-af98-c39705ce3966 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:39.589762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-2dd1b32c-4a48-49f0-af98-c39705ce3966 None None] MFA Rules not processed for user `57c88f16c7df40adb917f49804fda21f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:39.591048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2dd1b32c-4a48-49f0-af98-c39705ce3966 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:39.713867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1165/2333] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 4 01:11:39 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:39.725196 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ecb96c-843b-4b98-9eb2-370f59f330a6 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:39.726873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-12ecb96c-843b-4b98-9eb2-370f59f330a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:39.740234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ecb96c-843b-4b98-9eb2-370f59f330a6 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:39.743806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3e90fe-1eb2-4f9e-b113-75a5f79e65e2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '69153344fc4b45588bdbcecdfcfd7c3f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:39.744396 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3e90fe-1eb2-4f9e-b113-75a5f79e65e2 service glance] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:39.744676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3e90fe-1eb2-4f9e-b113-75a5f79e65e2 service glance] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:39.744952 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3e90fe-1eb2-4f9e-b113-75a5f79e65e2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:39.755626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3e90fe-1eb2-4f9e-b113-75a5f79e65e2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:39.757057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3e90fe-1eb2-4f9e-b113-75a5f79e65e2 service glance] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:39.768357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1169/2334] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:11:39 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:11:39.981820 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2c5169-d018-4ce6-a101-6821e3e3e204 None None] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:39.982099 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2c5169-d018-4ce6-a101-6821e3e3e204 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:39.982606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2c5169-d018-4ce6-a101-6821e3e3e204 None None] PATH_INFO: `/` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:39.984128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1166/2335] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 4 01:11:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:11:39.989054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828fe987-a0c0-4ac8-aeab-546936b096a6 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:39.989387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828fe987-a0c0-4ac8-aeab-546936b096a6 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:39.989661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828fe987-a0c0-4ac8-aeab-546936b096a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:40.009686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-828fe987-a0c0-4ac8-aeab-546936b096a6 None None] MFA Rules not processed for user `da4f44b86ca94adf959d9a41d915c902`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:40.010877 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-828fe987-a0c0-4ac8-aeab-546936b096a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:40.063197 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1170/2336] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 4 01:11:39 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:40.077941 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2c5169-d018-4ce6-a101-6821e3e3e204 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:40.080123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-bb2c5169-d018-4ce6-a101-6821e3e3e204 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:40.095996 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2c5169-d018-4ce6-a101-6821e3e3e204 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:40.100561 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0343c4f6-a6f8-4852-b748-8b0f1cc0717e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:40.104396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0343c4f6-a6f8-4852-b748-8b0f1cc0717e service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:40.104396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0343c4f6-a6f8-4852-b748-8b0f1cc0717e service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:40.104396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0343c4f6-a6f8-4852-b748-8b0f1cc0717e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:40.110330 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0343c4f6-a6f8-4852-b748-8b0f1cc0717e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:40.111777 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0343c4f6-a6f8-4852-b748-8b0f1cc0717e service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:40.121333 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1167/2337] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:11:40 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:11:40.209515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be4b260-187a-4aac-88df-e3412a6e1cb2 None None] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:40.209801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be4b260-187a-4aac-88df-e3412a6e1cb2 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:40.210075 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be4b260-187a-4aac-88df-e3412a6e1cb2 None None] PATH_INFO: `/` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:40.211432 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1171/2338] 10.222.0.42 () {54 vars in 921 bytes} [Sun Sep 4 01:11:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 04 01:11:40.218016 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f82e5ab-1c6a-49f6-9414-b29711a889aa None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:40.218308 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f82e5ab-1c6a-49f6-9414-b29711a889aa None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:40.218586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f82e5ab-1c6a-49f6-9414-b29711a889aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:40.243772 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-3f82e5ab-1c6a-49f6-9414-b29711a889aa None None] MFA Rules not processed for user `57c88f16c7df40adb917f49804fda21f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:40.244992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-3f82e5ab-1c6a-49f6-9414-b29711a889aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:40.255540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1168/2339] 10.222.0.42 () {58 vars in 1018 bytes} [Sun Sep 4 01:11:40 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:40.269296 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be4b260-187a-4aac-88df-e3412a6e1cb2 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:40.280500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be4b260-187a-4aac-88df-e3412a6e1cb2 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:40.284223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e76ad0-8548-4730-b170-7a1b222cd639 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:40.284817 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e76ad0-8548-4730-b170-7a1b222cd639 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:40.285117 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e76ad0-8548-4730-b170-7a1b222cd639 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:40.285395 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e76ad0-8548-4730-b170-7a1b222cd639 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:40.296673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e76ad0-8548-4730-b170-7a1b222cd639 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:40.298133 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e76ad0-8548-4730-b170-7a1b222cd639 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:40.308550 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1172/2340] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:11:40 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:11:43.220275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe69f58-28c9-4783-ae08-ffa5cef63f43 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:43.221004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe69f58-28c9-4783-ae08-ffa5cef63f43 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:43.221004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe69f58-28c9-4783-ae08-ffa5cef63f43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:43.241776 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-bbe69f58-28c9-4783-ae08-ffa5cef63f43 None None] MFA Rules not processed for user `70cef7c271e046a3b49230057cc2d28f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:43.242996 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-bbe69f58-28c9-4783-ae08-ffa5cef63f43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:43.300388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1169/2341] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 4 01:11:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:43.312996 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e76ad0-8548-4730-b170-7a1b222cd639 service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:43.314484 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-44e76ad0-8548-4730-b170-7a1b222cd639 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:43.386838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e76ad0-8548-4730-b170-7a1b222cd639 service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:43.392505 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354fabc-c4b2-4267-937c-5c6e5ff76e27 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '123ab6ff796c4e599ce1d6f4095bd898', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:43.393126 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354fabc-c4b2-4267-937c-5c6e5ff76e27 service swift] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:43.393411 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354fabc-c4b2-4267-937c-5c6e5ff76e27 service swift] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:43.393685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354fabc-c4b2-4267-937c-5c6e5ff76e27 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:43.401625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c354fabc-c4b2-4267-937c-5c6e5ff76e27 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:43.403204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c354fabc-c4b2-4267-937c-5c6e5ff76e27 service swift] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:43.411885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1173/2342] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 4 01:11:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 04 01:11:48.133503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe69f58-28c9-4783-ae08-ffa5cef63f43 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:48.168371 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:48.222702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe69f58-28c9-4783-ae08-ffa5cef63f43 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:48.227218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01a5f91-b577-403a-a51f-105b0bd57234 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:48.227838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01a5f91-b577-403a-a51f-105b0bd57234 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:48.228119 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01a5f91-b577-403a-a51f-105b0bd57234 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:48.228388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01a5f91-b577-403a-a51f-105b0bd57234 admin admin] PATH_INFO: `/v3/users/538a540a102842a99dfac6ae9264243e` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:48.229536 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01a5f91-b577-403a-a51f-105b0bd57234 admin admin] RBAC: Authorizing `identity:delete_user(user_id=538a540a102842a99dfac6ae9264243e)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:48.231487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01a5f91-b577-403a-a51f-105b0bd57234 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:48.416841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b01a5f91-b577-403a-a51f-105b0bd57234 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '538a540a102842a99dfac6ae9264243e'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:48.417560 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b01a5f91-b577-403a-a51f-105b0bd57234 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '538a540a102842a99dfac6ae9264243e'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:48.494595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b01a5f91-b577-403a-a51f-105b0bd57234 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '538a540a102842a99dfac6ae9264243e'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:48.515646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b01a5f91-b577-403a-a51f-105b0bd57234 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '538a540a102842a99dfac6ae9264243e'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:48.516730 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b01a5f91-b577-403a-a51f-105b0bd57234 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '538a540a102842a99dfac6ae9264243e'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:48.531267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1170/2343] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:11:48 2022] DELETE /identity/v3/users/538a540a102842a99dfac6ae9264243e => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:48.536181 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d465e65e-2401-4f03-95e6-1297d6a673c3 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:48.536675 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d465e65e-2401-4f03-95e6-1297d6a673c3 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:48.536760 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d465e65e-2401-4f03-95e6-1297d6a673c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:48.571828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-d465e65e-2401-4f03-95e6-1297d6a673c3 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:48.573065 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d465e65e-2401-4f03-95e6-1297d6a673c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:48.648804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1174/2344] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:48.658817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01a5f91-b577-403a-a51f-105b0bd57234 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:48.660306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b01a5f91-b577-403a-a51f-105b0bd57234 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:48.728234 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01a5f91-b577-403a-a51f-105b0bd57234 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:48.735058 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef42265a-50c3-4363-aaba-c5676f6437e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:48.736092 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef42265a-50c3-4363-aaba-c5676f6437e3 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:48.736533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef42265a-50c3-4363-aaba-c5676f6437e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:48.736979 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef42265a-50c3-4363-aaba-c5676f6437e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:48.749643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef42265a-50c3-4363-aaba-c5676f6437e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:48.751728 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef42265a-50c3-4363-aaba-c5676f6437e3 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:48.765307 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1171/2345] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:11:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:11:49.794433 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d465e65e-2401-4f03-95e6-1297d6a673c3 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:49.796540 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d465e65e-2401-4f03-95e6-1297d6a673c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:49.803839 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:49.817692 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d465e65e-2401-4f03-95e6-1297d6a673c3 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:49.835376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0a4425-5959-4fe1-b9a2-71f5a69dddd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:49.836557 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0a4425-5959-4fe1-b9a2-71f5a69dddd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:49.836927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0a4425-5959-4fe1-b9a2-71f5a69dddd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:49.837287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0a4425-5959-4fe1-b9a2-71f5a69dddd6 admin admin] PATH_INFO: `/v3/projects/57b717c160e64aca8571ae6cbeeee26c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:49.838642 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0a4425-5959-4fe1-b9a2-71f5a69dddd6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=57b717c160e64aca8571ae6cbeeee26c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:49.840783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0a4425-5959-4fe1-b9a2-71f5a69dddd6 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:50.014765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3c0a4425-5959-4fe1-b9a2-71f5a69dddd6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '57b717c160e64aca8571ae6cbeeee26c', 'request_id': 'req-3c0a4425-5959-4fe1-b9a2-71f5a69dddd6'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:50.020278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3c0a4425-5959-4fe1-b9a2-71f5a69dddd6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '57b717c160e64aca8571ae6cbeeee26c', 'request_id': 'req-3c0a4425-5959-4fe1-b9a2-71f5a69dddd6'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:50.026541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3c0a4425-5959-4fe1-b9a2-71f5a69dddd6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '57b717c160e64aca8571ae6cbeeee26c', 'request_id': 'req-3c0a4425-5959-4fe1-b9a2-71f5a69dddd6'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:50.093333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3c0a4425-5959-4fe1-b9a2-71f5a69dddd6 admin admin] The token cache is being invalidate because project 57b717c160e64aca8571ae6cbeeee26c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:11:50.093624 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3c0a4425-5959-4fe1-b9a2-71f5a69dddd6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:50.094249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3c0a4425-5959-4fe1-b9a2-71f5a69dddd6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:50.095736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1175/2346] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:11:49 2022] DELETE /identity/v3/projects/57b717c160e64aca8571ae6cbeeee26c => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:50.100663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ddc1f2-fe76-4f03-95d6-63f2baca09b4 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:50.101236 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ddc1f2-fe76-4f03-95d6-63f2baca09b4 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:50.101346 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ddc1f2-fe76-4f03-95d6-63f2baca09b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:50.122919 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-98ddc1f2-fe76-4f03-95d6-63f2baca09b4 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:50.125021 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-98ddc1f2-fe76-4f03-95d6-63f2baca09b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:50.163629 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:50.205891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1172/2347] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:50.213846 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a0b943-2b61-4af6-b38e-30fc26b83804 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:50.214186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a0b943-2b61-4af6-b38e-30fc26b83804 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:50.214408 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a0b943-2b61-4af6-b38e-30fc26b83804 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:50.239969 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-70a0b943-2b61-4af6-b38e-30fc26b83804 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:50.242008 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-70a0b943-2b61-4af6-b38e-30fc26b83804 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:50.253864 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1176/2348] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:50.256468 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ddc1f2-fe76-4f03-95d6-63f2baca09b4 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:50.268143 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ddc1f2-fe76-4f03-95d6-63f2baca09b4 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:50.276278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5245ffb-3b2f-4807-80c0-6c6b32aa07f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:50.277267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5245ffb-3b2f-4807-80c0-6c6b32aa07f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:50.284109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5245ffb-3b2f-4807-80c0-6c6b32aa07f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:50.284109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5245ffb-3b2f-4807-80c0-6c6b32aa07f7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:50.284109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5245ffb-3b2f-4807-80c0-6c6b32aa07f7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:11:50.284109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5245ffb-3b2f-4807-80c0-6c6b32aa07f7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:50.284109 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5245ffb-3b2f-4807-80c0-6c6b32aa07f7 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:50.296333 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1173/2349] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:11:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:11:50.300377 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a0b943-2b61-4af6-b38e-30fc26b83804 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:50.309895 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a0b943-2b61-4af6-b38e-30fc26b83804 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:50.315846 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9f88c3-2afe-4d69-b02b-b24adac6ec7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:50.316450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9f88c3-2afe-4d69-b02b-b24adac6ec7a admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:50.316726 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9f88c3-2afe-4d69-b02b-b24adac6ec7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:50.316995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9f88c3-2afe-4d69-b02b-b24adac6ec7a admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:50.317754 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9f88c3-2afe-4d69-b02b-b24adac6ec7a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:50.319675 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9f88c3-2afe-4d69-b02b-b24adac6ec7a admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:50.417169 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1177/2350] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:11:50 2022] POST /identity/v3/projects => generated 375 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:11:50.417400 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5245ffb-3b2f-4807-80c0-6c6b32aa07f7 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:50.429838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5245ffb-3b2f-4807-80c0-6c6b32aa07f7 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:50.542172 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d3e3d4-fe0c-4bb8-90ca-efd92176f440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:50.596924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d3e3d4-fe0c-4bb8-90ca-efd92176f440 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:50.596924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d3e3d4-fe0c-4bb8-90ca-efd92176f440 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:50.596924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d3e3d4-fe0c-4bb8-90ca-efd92176f440 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:50.596924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d3e3d4-fe0c-4bb8-90ca-efd92176f440 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:50.596924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d3e3d4-fe0c-4bb8-90ca-efd92176f440 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:50.705451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1174/2351] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:11:50 2022] POST /identity/v3/users => generated 288 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:11:50.706140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9f88c3-2afe-4d69-b02b-b24adac6ec7a admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:50.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9f88c3-2afe-4d69-b02b-b24adac6ec7a admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:50.722522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3ae7bc-d607-463b-8b35-e5306505ac1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:50.723119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3ae7bc-d607-463b-8b35-e5306505ac1e admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:50.723396 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3ae7bc-d607-463b-8b35-e5306505ac1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:50.723665 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3ae7bc-d607-463b-8b35-e5306505ac1e admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:50.724311 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3ae7bc-d607-463b-8b35-e5306505ac1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:50.725949 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3ae7bc-d607-463b-8b35-e5306505ac1e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:50.743382 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1178/2352] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:50 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:11:50.744562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d3e3d4-fe0c-4bb8-90ca-efd92176f440 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:50.767145 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d3e3d4-fe0c-4bb8-90ca-efd92176f440 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:50.775280 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa499cc2-236a-4bff-97b4-451c75b23160 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:50.775280 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa499cc2-236a-4bff-97b4-451c75b23160 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:50.775280 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa499cc2-236a-4bff-97b4-451c75b23160 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:50.775280 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa499cc2-236a-4bff-97b4-451c75b23160 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:50.775679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa499cc2-236a-4bff-97b4-451c75b23160 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:50.775679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa499cc2-236a-4bff-97b4-451c75b23160 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:50.800351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1175/2353] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:50 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:11:50.802953 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3ae7bc-d607-463b-8b35-e5306505ac1e admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:50.818957 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3ae7bc-d607-463b-8b35-e5306505ac1e admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:50.826046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09746e35-dfbd-4fce-9e85-35f7be22f712 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:50.826979 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09746e35-dfbd-4fce-9e85-35f7be22f712 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:50.827435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09746e35-dfbd-4fce-9e85-35f7be22f712 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:50.827873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09746e35-dfbd-4fce-9e85-35f7be22f712 admin admin] PATH_INFO: `/v3/projects/8758a3c4d7334a51a1615b94f7cf10b2/users/482edb0cead34e36879a65c68812577f/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:50.845496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09746e35-dfbd-4fce-9e85-35f7be22f712 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8758a3c4d7334a51a1615b94f7cf10b2, user_id=482edb0cead34e36879a65c68812577f, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:50.848388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09746e35-dfbd-4fce-9e85-35f7be22f712 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:50.987585 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1179/2354] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:11:50 2022] PUT /identity/v3/projects/8758a3c4d7334a51a1615b94f7cf10b2/users/482edb0cead34e36879a65c68812577f/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:50.999123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7998d7f-8652-463c-b218-4332dc6eafe0 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:50.999409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7998d7f-8652-463c-b218-4332dc6eafe0 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:50.999682 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7998d7f-8652-463c-b218-4332dc6eafe0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:51.026747 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-b7998d7f-8652-463c-b218-4332dc6eafe0 None None] MFA Rules not processed for user `482edb0cead34e36879a65c68812577f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:51.027958 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b7998d7f-8652-463c-b218-4332dc6eafe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:51.093373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1176/2355] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:50 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:51.097903 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09746e35-dfbd-4fce-9e85-35f7be22f712 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:51.164507 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09746e35-dfbd-4fce-9e85-35f7be22f712 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:51.169668 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528464be-cc31-45d6-b299-fad1fd512d01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:51.170907 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528464be-cc31-45d6-b299-fad1fd512d01 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:51.171185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528464be-cc31-45d6-b299-fad1fd512d01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:51.171447 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528464be-cc31-45d6-b299-fad1fd512d01 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:51.172090 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528464be-cc31-45d6-b299-fad1fd512d01 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:51.174152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528464be-cc31-45d6-b299-fad1fd512d01 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:51.245276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1180/2356] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:11:51 2022] POST /identity/v3/projects => generated 373 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:11:51.249579 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7998d7f-8652-463c-b218-4332dc6eafe0 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:51.260492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7998d7f-8652-463c-b218-4332dc6eafe0 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:51.288433 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:51.317937 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4c095f-deaa-466c-977d-f153a2167fbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:51.318543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4c095f-deaa-466c-977d-f153a2167fbc admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:51.318820 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4c095f-deaa-466c-977d-f153a2167fbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:51.319091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4c095f-deaa-466c-977d-f153a2167fbc admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:51.319709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4c095f-deaa-466c-977d-f153a2167fbc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:51.321640 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4c095f-deaa-466c-977d-f153a2167fbc admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:51.408410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1177/2357] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:11:51 2022] POST /identity/v3/users => generated 293 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:11:51.412419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528464be-cc31-45d6-b299-fad1fd512d01 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:51.423275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528464be-cc31-45d6-b299-fad1fd512d01 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:51.427993 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2586c94e-e671-4fb4-bce5-5b723a17176b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:51.428618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2586c94e-e671-4fb4-bce5-5b723a17176b admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:51.428996 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2586c94e-e671-4fb4-bce5-5b723a17176b admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:51.429229 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2586c94e-e671-4fb4-bce5-5b723a17176b admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:51.429871 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2586c94e-e671-4fb4-bce5-5b723a17176b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:51.431471 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2586c94e-e671-4fb4-bce5-5b723a17176b admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:51.432835 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:51.442875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1181/2358] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:11:51.447834 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4c095f-deaa-466c-977d-f153a2167fbc admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:51.460020 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4c095f-deaa-466c-977d-f153a2167fbc admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:51.464862 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691b32c1-018b-47c2-a3f5-5d7c07021c66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:51.465540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691b32c1-018b-47c2-a3f5-5d7c07021c66 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:51.465820 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691b32c1-018b-47c2-a3f5-5d7c07021c66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:51.466091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691b32c1-018b-47c2-a3f5-5d7c07021c66 admin admin] PATH_INFO: `/v3/projects/b8d506bb2b664efaab15c7efa82d40d9/users/dce70d16f1f54b8d943375667e3e1b05/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:51.488550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691b32c1-018b-47c2-a3f5-5d7c07021c66 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8d506bb2b664efaab15c7efa82d40d9, user_id=dce70d16f1f54b8d943375667e3e1b05, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:51.491494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691b32c1-018b-47c2-a3f5-5d7c07021c66 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:51.581979 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1178/2359] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:11:51 2022] PUT /identity/v3/projects/b8d506bb2b664efaab15c7efa82d40d9/users/dce70d16f1f54b8d943375667e3e1b05/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:51.583548 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2586c94e-e671-4fb4-bce5-5b723a17176b admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:51.614647 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2586c94e-e671-4fb4-bce5-5b723a17176b admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:51.729714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82013278-db93-43c0-b6a7-1387f5f98167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:51.730575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82013278-db93-43c0-b6a7-1387f5f98167 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:51.731076 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82013278-db93-43c0-b6a7-1387f5f98167 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:51.731577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82013278-db93-43c0-b6a7-1387f5f98167 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:51.732584 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82013278-db93-43c0-b6a7-1387f5f98167 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:51.735183 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82013278-db93-43c0-b6a7-1387f5f98167 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:51.750338 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1182/2360] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:51 2022] GET /identity/v3/roles => generated 1420 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:11:51.754213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691b32c1-018b-47c2-a3f5-5d7c07021c66 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:51.765640 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691b32c1-018b-47c2-a3f5-5d7c07021c66 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:51.771189 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c971c45-f7ff-4e82-b634-0d696e154980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:51.771984 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c971c45-f7ff-4e82-b634-0d696e154980 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:51.772370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c971c45-f7ff-4e82-b634-0d696e154980 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:51.772778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c971c45-f7ff-4e82-b634-0d696e154980 admin admin] PATH_INFO: `/v3/projects/b8d506bb2b664efaab15c7efa82d40d9/users/dce70d16f1f54b8d943375667e3e1b05/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:51.774843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c971c45-f7ff-4e82-b634-0d696e154980 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8d506bb2b664efaab15c7efa82d40d9, user_id=dce70d16f1f54b8d943375667e3e1b05, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:51.777624 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c971c45-f7ff-4e82-b634-0d696e154980 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:51.790170 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1179/2361] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:11:51 2022] PUT /identity/v3/projects/b8d506bb2b664efaab15c7efa82d40d9/users/dce70d16f1f54b8d943375667e3e1b05/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:11:51.858286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6afcd4-60d6-4b2c-837a-5de9bd15bed3 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:51.858884 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6afcd4-60d6-4b2c-837a-5de9bd15bed3 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:51.858989 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6afcd4-60d6-4b2c-837a-5de9bd15bed3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:51.887486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-1b6afcd4-60d6-4b2c-837a-5de9bd15bed3 None None] MFA Rules not processed for user `dce70d16f1f54b8d943375667e3e1b05`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:51.888727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-1b6afcd4-60d6-4b2c-837a-5de9bd15bed3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:51.962272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1183/2362] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:51 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:51.973865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c971c45-f7ff-4e82-b634-0d696e154980 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:51.975574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-1c971c45-f7ff-4e82-b634-0d696e154980 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:52.041484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c971c45-f7ff-4e82-b634-0d696e154980 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:52.045363 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2ca40c-baf8-4d1f-aeeb-172634907f8d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:52.046053 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2ca40c-baf8-4d1f-aeeb-172634907f8d service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:52.046432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2ca40c-baf8-4d1f-aeeb-172634907f8d service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:52.046686 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2ca40c-baf8-4d1f-aeeb-172634907f8d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:52.114970 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2ca40c-baf8-4d1f-aeeb-172634907f8d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:52.116747 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2ca40c-baf8-4d1f-aeeb-172634907f8d service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:52.126382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1180/2363] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:11:51 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:11:55.326123 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6afcd4-60d6-4b2c-837a-5de9bd15bed3 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:55.387928 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6afcd4-60d6-4b2c-837a-5de9bd15bed3 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:55.392558 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fed20e9-a489-4de7-b4ae-07354b765f43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:55.393211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fed20e9-a489-4de7-b4ae-07354b765f43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:55.393487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fed20e9-a489-4de7-b4ae-07354b765f43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:55.393756 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fed20e9-a489-4de7-b4ae-07354b765f43 admin admin] PATH_INFO: `/v3/users/482edb0cead34e36879a65c68812577f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:55.394879 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fed20e9-a489-4de7-b4ae-07354b765f43 admin admin] RBAC: Authorizing `identity:delete_user(user_id=482edb0cead34e36879a65c68812577f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:55.396807 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fed20e9-a489-4de7-b4ae-07354b765f43 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:55.418535 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:55.582772 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-8fed20e9-a489-4de7-b4ae-07354b765f43 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '482edb0cead34e36879a65c68812577f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:55.648403 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-8fed20e9-a489-4de7-b4ae-07354b765f43 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '482edb0cead34e36879a65c68812577f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:55.660965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-8fed20e9-a489-4de7-b4ae-07354b765f43 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '482edb0cead34e36879a65c68812577f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:55.661639 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-8fed20e9-a489-4de7-b4ae-07354b765f43 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '482edb0cead34e36879a65c68812577f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:55.668520 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-8fed20e9-a489-4de7-b4ae-07354b765f43 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '482edb0cead34e36879a65c68812577f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:55.676861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1184/2364] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:11:55 2022] DELETE /identity/v3/users/482edb0cead34e36879a65c68812577f => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:55.681971 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80862ea9-b572-4cbe-90f2-98725102fb12 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:55.682519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80862ea9-b572-4cbe-90f2-98725102fb12 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:55.682519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80862ea9-b572-4cbe-90f2-98725102fb12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:55.702650 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-80862ea9-b572-4cbe-90f2-98725102fb12 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:55.703884 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-80862ea9-b572-4cbe-90f2-98725102fb12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:55.733734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:55.763895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1181/2365] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:55.773955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fed20e9-a489-4de7-b4ae-07354b765f43 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:55.775774 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-8fed20e9-a489-4de7-b4ae-07354b765f43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:55.841632 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fed20e9-a489-4de7-b4ae-07354b765f43 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:55.846266 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e3e690-d81b-4df9-8e51-b195bb5cc5db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:55.846875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e3e690-d81b-4df9-8e51-b195bb5cc5db service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:55.847229 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e3e690-d81b-4df9-8e51-b195bb5cc5db service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:55.847588 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e3e690-d81b-4df9-8e51-b195bb5cc5db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:55.856923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e3e690-d81b-4df9-8e51-b195bb5cc5db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:55.858446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e3e690-d81b-4df9-8e51-b195bb5cc5db service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:55.868382 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1185/2366] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:11:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:11:56.244479 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80862ea9-b572-4cbe-90f2-98725102fb12 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:56.245971 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-80862ea9-b572-4cbe-90f2-98725102fb12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:56.260024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80862ea9-b572-4cbe-90f2-98725102fb12 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:56.264916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a618146d-3212-4115-aaba-b3ba7040445b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:56.265591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a618146d-3212-4115-aaba-b3ba7040445b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:56.265871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a618146d-3212-4115-aaba-b3ba7040445b admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:56.266143 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a618146d-3212-4115-aaba-b3ba7040445b admin admin] PATH_INFO: `/v3/projects/8758a3c4d7334a51a1615b94f7cf10b2` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:56.267283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a618146d-3212-4115-aaba-b3ba7040445b admin admin] RBAC: Authorizing `identity:delete_project(project_id=8758a3c4d7334a51a1615b94f7cf10b2)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:56.269209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a618146d-3212-4115-aaba-b3ba7040445b admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:56.377333 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a618146d-3212-4115-aaba-b3ba7040445b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8758a3c4d7334a51a1615b94f7cf10b2', 'request_id': 'req-a618146d-3212-4115-aaba-b3ba7040445b'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:56.383172 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a618146d-3212-4115-aaba-b3ba7040445b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8758a3c4d7334a51a1615b94f7cf10b2', 'request_id': 'req-a618146d-3212-4115-aaba-b3ba7040445b'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:56.469691 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a618146d-3212-4115-aaba-b3ba7040445b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8758a3c4d7334a51a1615b94f7cf10b2', 'request_id': 'req-a618146d-3212-4115-aaba-b3ba7040445b'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:56.527423 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a618146d-3212-4115-aaba-b3ba7040445b admin admin] The token cache is being invalidate because project 8758a3c4d7334a51a1615b94f7cf10b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:11:56.527423 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a618146d-3212-4115-aaba-b3ba7040445b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:56.527423 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a618146d-3212-4115-aaba-b3ba7040445b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:56.527423 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1182/2367] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:11:56 2022] DELETE /identity/v3/projects/8758a3c4d7334a51a1615b94f7cf10b2 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:56.528274 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e3e690-d81b-4df9-8e51-b195bb5cc5db service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:56.528274 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e3e690-d81b-4df9-8e51-b195bb5cc5db service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:56.581036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0beb891b-cbee-4233-92ae-037e69f84552 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:56.581710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0beb891b-cbee-4233-92ae-037e69f84552 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:56.581991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0beb891b-cbee-4233-92ae-037e69f84552 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:56.582264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0beb891b-cbee-4233-92ae-037e69f84552 admin admin] PATH_INFO: `/v3/users/dce70d16f1f54b8d943375667e3e1b05` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:56.583395 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0beb891b-cbee-4233-92ae-037e69f84552 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dce70d16f1f54b8d943375667e3e1b05)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:56.585360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0beb891b-cbee-4233-92ae-037e69f84552 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:56.771801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:56.776773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0beb891b-cbee-4233-92ae-037e69f84552 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dce70d16f1f54b8d943375667e3e1b05'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:56.843906 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0beb891b-cbee-4233-92ae-037e69f84552 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dce70d16f1f54b8d943375667e3e1b05'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:56.858434 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0beb891b-cbee-4233-92ae-037e69f84552 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dce70d16f1f54b8d943375667e3e1b05'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:56.859179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0beb891b-cbee-4233-92ae-037e69f84552 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dce70d16f1f54b8d943375667e3e1b05'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:56.866828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0beb891b-cbee-4233-92ae-037e69f84552 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dce70d16f1f54b8d943375667e3e1b05'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:56.872698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1186/2368] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:11:56 2022] DELETE /identity/v3/users/dce70d16f1f54b8d943375667e3e1b05 => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:57.371030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a618146d-3212-4115-aaba-b3ba7040445b admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:57.376208 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a618146d-3212-4115-aaba-b3ba7040445b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:57.389238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a618146d-3212-4115-aaba-b3ba7040445b admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:57.449808 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ee31fc-2c2d-4ff4-bf0e-66796f177066 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:57.450898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ee31fc-2c2d-4ff4-bf0e-66796f177066 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:57.450898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ee31fc-2c2d-4ff4-bf0e-66796f177066 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:57.451128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ee31fc-2c2d-4ff4-bf0e-66796f177066 admin admin] PATH_INFO: `/v3/projects/b8d506bb2b664efaab15c7efa82d40d9` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:57.452119 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ee31fc-2c2d-4ff4-bf0e-66796f177066 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8d506bb2b664efaab15c7efa82d40d9)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:57.454179 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ee31fc-2c2d-4ff4-bf0e-66796f177066 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:57.552339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-04ee31fc-2c2d-4ff4-bf0e-66796f177066 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8d506bb2b664efaab15c7efa82d40d9', 'request_id': 'req-04ee31fc-2c2d-4ff4-bf0e-66796f177066'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:57.558779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-04ee31fc-2c2d-4ff4-bf0e-66796f177066 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8d506bb2b664efaab15c7efa82d40d9', 'request_id': 'req-04ee31fc-2c2d-4ff4-bf0e-66796f177066'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:57.562671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:57.746220 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-04ee31fc-2c2d-4ff4-bf0e-66796f177066 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8d506bb2b664efaab15c7efa82d40d9', 'request_id': 'req-04ee31fc-2c2d-4ff4-bf0e-66796f177066'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:57.766552 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-04ee31fc-2c2d-4ff4-bf0e-66796f177066 admin admin] The token cache is being invalidate because project b8d506bb2b664efaab15c7efa82d40d9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:11:57.766552 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-04ee31fc-2c2d-4ff4-bf0e-66796f177066 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:57.766552 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-04ee31fc-2c2d-4ff4-bf0e-66796f177066 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:11:57.767518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1183/2369] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:11:57 2022] DELETE /identity/v3/projects/b8d506bb2b664efaab15c7efa82d40d9 => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:57.772262 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8058d111-28e0-4e7b-962b-0388697c298f None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:57.772954 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8058d111-28e0-4e7b-962b-0388697c298f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:57.772954 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8058d111-28e0-4e7b-962b-0388697c298f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:57.801372 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-8058d111-28e0-4e7b-962b-0388697c298f None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:57.802964 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-8058d111-28e0-4e7b-962b-0388697c298f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:57.886914 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1187/2370] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:57.892262 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21856e4-c148-4b4e-b3a3-12a940c3f964 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:57.892573 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21856e4-c148-4b4e-b3a3-12a940c3f964 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:57.892895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21856e4-c148-4b4e-b3a3-12a940c3f964 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:57.916289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-b21856e4-c148-4b4e-b3a3-12a940c3f964 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:57.917561 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b21856e4-c148-4b4e-b3a3-12a940c3f964 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:57.927423 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1184/2371] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:57.935539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8058d111-28e0-4e7b-962b-0388697c298f None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:57.945753 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8058d111-28e0-4e7b-962b-0388697c298f None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:57.950292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c39b4e-c9a4-4843-9e1a-c15f9c795768 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:57.950881 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c39b4e-c9a4-4843-9e1a-c15f9c795768 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:57.951181 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c39b4e-c9a4-4843-9e1a-c15f9c795768 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:57.951470 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c39b4e-c9a4-4843-9e1a-c15f9c795768 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:57.952104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c39b4e-c9a4-4843-9e1a-c15f9c795768 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:11:57.952398 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c39b4e-c9a4-4843-9e1a-c15f9c795768 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:57.954018 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c39b4e-c9a4-4843-9e1a-c15f9c795768 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:57.965575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1188/2372] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:11:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:11:57.970518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21856e4-c148-4b4e-b3a3-12a940c3f964 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:57.980082 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21856e4-c148-4b4e-b3a3-12a940c3f964 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:57.985531 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287faefc-19c5-4337-8cf3-84237174be08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:57.986132 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287faefc-19c5-4337-8cf3-84237174be08 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:57.986415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287faefc-19c5-4337-8cf3-84237174be08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:57.986690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287faefc-19c5-4337-8cf3-84237174be08 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:57.987336 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287faefc-19c5-4337-8cf3-84237174be08 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:57.989253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287faefc-19c5-4337-8cf3-84237174be08 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:58.089798 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1185/2373] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:11:57 2022] POST /identity/v3/projects => generated 385 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:11:58.094849 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c39b4e-c9a4-4843-9e1a-c15f9c795768 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:58.105236 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c39b4e-c9a4-4843-9e1a-c15f9c795768 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:58.163120 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1e9902-7df4-444e-bce3-53437505e3e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:58.163645 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1e9902-7df4-444e-bce3-53437505e3e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:58.163931 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1e9902-7df4-444e-bce3-53437505e3e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:58.164206 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1e9902-7df4-444e-bce3-53437505e3e5 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:58.164830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1e9902-7df4-444e-bce3-53437505e3e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:58.166813 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1e9902-7df4-444e-bce3-53437505e3e5 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:58.248576 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1189/2374] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:11:58 2022] POST /identity/v3/users => generated 293 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:11:58.253000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287faefc-19c5-4337-8cf3-84237174be08 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:58.263246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287faefc-19c5-4337-8cf3-84237174be08 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:58.268606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d719b9e8-eeb5-47d2-81dd-493f99f5d93c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:58.274883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d719b9e8-eeb5-47d2-81dd-493f99f5d93c admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:58.274883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d719b9e8-eeb5-47d2-81dd-493f99f5d93c admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:58.274883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d719b9e8-eeb5-47d2-81dd-493f99f5d93c admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:58.274883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d719b9e8-eeb5-47d2-81dd-493f99f5d93c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:58.274883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d719b9e8-eeb5-47d2-81dd-493f99f5d93c admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:58.282956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1186/2375] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:11:58.287019 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1e9902-7df4-444e-bce3-53437505e3e5 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:58.296999 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1e9902-7df4-444e-bce3-53437505e3e5 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:58.301647 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be71e98-ed7d-4370-9392-63bb06c08a42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:58.302239 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be71e98-ed7d-4370-9392-63bb06c08a42 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:58.302520 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be71e98-ed7d-4370-9392-63bb06c08a42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:58.302796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be71e98-ed7d-4370-9392-63bb06c08a42 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:58.303437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be71e98-ed7d-4370-9392-63bb06c08a42 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:58.305033 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be71e98-ed7d-4370-9392-63bb06c08a42 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:58.315514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1190/2376] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:58 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:11:58.319562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d719b9e8-eeb5-47d2-81dd-493f99f5d93c admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:58.329949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d719b9e8-eeb5-47d2-81dd-493f99f5d93c admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:58.334542 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bc1bf8-5d3b-45ef-9ca8-c97e50380cbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:58.335174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bc1bf8-5d3b-45ef-9ca8-c97e50380cbd admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:58.335450 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bc1bf8-5d3b-45ef-9ca8-c97e50380cbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:58.335742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bc1bf8-5d3b-45ef-9ca8-c97e50380cbd admin admin] PATH_INFO: `/v3/projects/4a9de8cefc614e71b43a8ccead0d671a/users/5608e90dc7bf4ca89b4f1127a5f14088/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:58.350545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bc1bf8-5d3b-45ef-9ca8-c97e50380cbd admin admin] RBAC: Authorizing `identity:create_grant(project_id=4a9de8cefc614e71b43a8ccead0d671a, user_id=5608e90dc7bf4ca89b4f1127a5f14088, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:58.353185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bc1bf8-5d3b-45ef-9ca8-c97e50380cbd admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:58.426272 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1187/2377] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:11:58 2022] PUT /identity/v3/projects/4a9de8cefc614e71b43a8ccead0d671a/users/5608e90dc7bf4ca89b4f1127a5f14088/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:11:58.436685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30549fb-8d2b-4bb8-a1fe-f8501153053b None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:58.437211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30549fb-8d2b-4bb8-a1fe-f8501153053b None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:58.437342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30549fb-8d2b-4bb8-a1fe-f8501153053b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:58.469889 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-f30549fb-8d2b-4bb8-a1fe-f8501153053b None None] MFA Rules not processed for user `5608e90dc7bf4ca89b4f1127a5f14088`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:58.469889 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-f30549fb-8d2b-4bb8-a1fe-f8501153053b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:58.481310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:58.536943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1191/2378] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:58 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:58.541606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bc1bf8-5d3b-45ef-9ca8-c97e50380cbd admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:58.606888 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bc1bf8-5d3b-45ef-9ca8-c97e50380cbd admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:58.611704 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295729ef-082a-4701-b588-5079f5e2372f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:58.612307 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295729ef-082a-4701-b588-5079f5e2372f admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:58.612583 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295729ef-082a-4701-b588-5079f5e2372f admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:58.612854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295729ef-082a-4701-b588-5079f5e2372f admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:58.613518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295729ef-082a-4701-b588-5079f5e2372f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:58.615679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295729ef-082a-4701-b588-5079f5e2372f admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:58.731968 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1188/2379] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:11:58 2022] POST /identity/v3/projects => generated 385 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:11:58.737092 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30549fb-8d2b-4bb8-a1fe-f8501153053b None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:58.749003 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30549fb-8d2b-4bb8-a1fe-f8501153053b None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:58.831790 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef99e16-a5fe-496c-8fd0-46e14a8db6f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:58.832404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef99e16-a5fe-496c-8fd0-46e14a8db6f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:58.832670 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef99e16-a5fe-496c-8fd0-46e14a8db6f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:58.833035 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef99e16-a5fe-496c-8fd0-46e14a8db6f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:58.833666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef99e16-a5fe-496c-8fd0-46e14a8db6f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:58.835593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef99e16-a5fe-496c-8fd0-46e14a8db6f1 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:58.938124 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1192/2380] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:11:58 2022] POST /identity/v3/users => generated 293 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:11:58.939455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295729ef-082a-4701-b588-5079f5e2372f admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:58.950499 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295729ef-082a-4701-b588-5079f5e2372f admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:58.955762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3532499c-a1f7-4dd0-880a-ce825ed4353a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:58.957054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3532499c-a1f7-4dd0-880a-ce825ed4353a admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:58.957054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3532499c-a1f7-4dd0-880a-ce825ed4353a admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:58.957054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3532499c-a1f7-4dd0-880a-ce825ed4353a admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:58.957719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3532499c-a1f7-4dd0-880a-ce825ed4353a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:58.959311 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3532499c-a1f7-4dd0-880a-ce825ed4353a admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:58.970967 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1189/2381] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:11:58.975095 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef99e16-a5fe-496c-8fd0-46e14a8db6f1 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:58.986099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef99e16-a5fe-496c-8fd0-46e14a8db6f1 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:58.991581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537362d3-aad2-496f-a06f-b11f6dcf5ff9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:58.991911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537362d3-aad2-496f-a06f-b11f6dcf5ff9 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:58.992256 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537362d3-aad2-496f-a06f-b11f6dcf5ff9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:58.992603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537362d3-aad2-496f-a06f-b11f6dcf5ff9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:58.993396 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537362d3-aad2-496f-a06f-b11f6dcf5ff9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:58.997409 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-537362d3-aad2-496f-a06f-b11f6dcf5ff9 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:59.009773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1193/2382] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:58 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:11:59.014665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3532499c-a1f7-4dd0-880a-ce825ed4353a admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:59.026032 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3532499c-a1f7-4dd0-880a-ce825ed4353a admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:59.032075 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b3ac86-341b-4d07-863e-48bb54cfe0c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:59.032741 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b3ac86-341b-4d07-863e-48bb54cfe0c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:59.033024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b3ac86-341b-4d07-863e-48bb54cfe0c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:59.033335 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b3ac86-341b-4d07-863e-48bb54cfe0c6 admin admin] PATH_INFO: `/v3/projects/7b00179d801c40fd8d01bcb03cf3f14b/users/baf4962efd214558997fac219f51950a/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:59.049733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b3ac86-341b-4d07-863e-48bb54cfe0c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b00179d801c40fd8d01bcb03cf3f14b, user_id=baf4962efd214558997fac219f51950a, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:59.052402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b3ac86-341b-4d07-863e-48bb54cfe0c6 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:59.111711 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1190/2383] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:11:59 2022] PUT /identity/v3/projects/7b00179d801c40fd8d01bcb03cf3f14b/users/baf4962efd214558997fac219f51950a/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:59.124187 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bd02ad-60fc-4a4c-a9a7-e52bfe3ddd4f None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:59.124527 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bd02ad-60fc-4a4c-a9a7-e52bfe3ddd4f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:59.124808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bd02ad-60fc-4a4c-a9a7-e52bfe3ddd4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:59.155166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-49bd02ad-60fc-4a4c-a9a7-e52bfe3ddd4f None None] MFA Rules not processed for user `baf4962efd214558997fac219f51950a`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:11:59.156546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-49bd02ad-60fc-4a4c-a9a7-e52bfe3ddd4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:11:59.222858 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1194/2384] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:59 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:11:59.227620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b3ac86-341b-4d07-863e-48bb54cfe0c6 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:59.267975 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:DELETE=1,keystone:SELECT=95,keystone:INSERT=4 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:59.291039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b3ac86-341b-4d07-863e-48bb54cfe0c6 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:59.296951 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda2f235-e1e1-4b98-a642-e49a6ffd3ed4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:59.297581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda2f235-e1e1-4b98-a642-e49a6ffd3ed4 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:59.297857 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda2f235-e1e1-4b98-a642-e49a6ffd3ed4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:59.298127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda2f235-e1e1-4b98-a642-e49a6ffd3ed4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:59.298765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda2f235-e1e1-4b98-a642-e49a6ffd3ed4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:59.300654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda2f235-e1e1-4b98-a642-e49a6ffd3ed4 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:59.374898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1191/2385] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:11:59 2022] POST /identity/v3/projects => generated 383 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:11:59.379024 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bd02ad-60fc-4a4c-a9a7-e52bfe3ddd4f None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:59.389815 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bd02ad-60fc-4a4c-a9a7-e52bfe3ddd4f None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:59.408027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:11:59.445954 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24175a9-9a51-4818-a5fa-0a7170691416 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:59.446557 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24175a9-9a51-4818-a5fa-0a7170691416 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:59.446837 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24175a9-9a51-4818-a5fa-0a7170691416 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:59.447110 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24175a9-9a51-4818-a5fa-0a7170691416 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:59.447751 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24175a9-9a51-4818-a5fa-0a7170691416 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:59.449732 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24175a9-9a51-4818-a5fa-0a7170691416 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:59.562870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1195/2386] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:11:59 2022] POST /identity/v3/users => generated 298 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:11:59.567160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda2f235-e1e1-4b98-a642-e49a6ffd3ed4 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:59.578231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda2f235-e1e1-4b98-a642-e49a6ffd3ed4 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:59.582912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617b4f4f-890c-4be0-b7e8-7f6347ad7504 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:59.583503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617b4f4f-890c-4be0-b7e8-7f6347ad7504 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:59.583785 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617b4f4f-890c-4be0-b7e8-7f6347ad7504 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:59.584054 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617b4f4f-890c-4be0-b7e8-7f6347ad7504 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:59.584695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617b4f4f-890c-4be0-b7e8-7f6347ad7504 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:59.586332 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617b4f4f-890c-4be0-b7e8-7f6347ad7504 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:59.596977 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1192/2387] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:59 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:11:59.600960 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24175a9-9a51-4818-a5fa-0a7170691416 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:59.610937 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24175a9-9a51-4818-a5fa-0a7170691416 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:59.615857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f8feba-cc3a-4943-af91-82410944978f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:59.616493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f8feba-cc3a-4943-af91-82410944978f admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:59.616767 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f8feba-cc3a-4943-af91-82410944978f admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:59.617032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f8feba-cc3a-4943-af91-82410944978f admin admin] PATH_INFO: `/v3/projects/b8b43148935f46118bf416336b1abdbe/users/e7ff3c53fa9b43afa9e77e467bd8dd58/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:59.632187 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f8feba-cc3a-4943-af91-82410944978f admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8b43148935f46118bf416336b1abdbe, user_id=e7ff3c53fa9b43afa9e77e467bd8dd58, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:59.634902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f8feba-cc3a-4943-af91-82410944978f admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:59.706739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1196/2388] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:11:59 2022] PUT /identity/v3/projects/b8b43148935f46118bf416336b1abdbe/users/e7ff3c53fa9b43afa9e77e467bd8dd58/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:59.710733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617b4f4f-890c-4be0-b7e8-7f6347ad7504 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:59.722411 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617b4f4f-890c-4be0-b7e8-7f6347ad7504 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:59.783835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b8233e-d558-4669-a008-f64d30fa29f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:59.784452 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b8233e-d558-4669-a008-f64d30fa29f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:59.784728 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b8233e-d558-4669-a008-f64d30fa29f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:59.784995 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b8233e-d558-4669-a008-f64d30fa29f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:59.785668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b8233e-d558-4669-a008-f64d30fa29f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:59.787240 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b8233e-d558-4669-a008-f64d30fa29f7 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:59.797878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1193/2389] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:11:59 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:11:59.802314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f8feba-cc3a-4943-af91-82410944978f admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:11:59.813238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f8feba-cc3a-4943-af91-82410944978f admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:11:59.817708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304b3e47-2ea2-40d7-9ac7-aed3ba62823b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:11:59.818405 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304b3e47-2ea2-40d7-9ac7-aed3ba62823b admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:59.818685 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304b3e47-2ea2-40d7-9ac7-aed3ba62823b admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:59.818957 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304b3e47-2ea2-40d7-9ac7-aed3ba62823b admin admin] PATH_INFO: `/v3/projects/b8b43148935f46118bf416336b1abdbe/users/e7ff3c53fa9b43afa9e77e467bd8dd58/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:11:59.820837 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304b3e47-2ea2-40d7-9ac7-aed3ba62823b admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8b43148935f46118bf416336b1abdbe, user_id=e7ff3c53fa9b43afa9e77e467bd8dd58, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:11:59.823531 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304b3e47-2ea2-40d7-9ac7-aed3ba62823b admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:11:59.834813 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1197/2390] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:11:59 2022] PUT /identity/v3/projects/b8b43148935f46118bf416336b1abdbe/users/e7ff3c53fa9b43afa9e77e467bd8dd58/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:11:59.845816 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e567edb-1b07-4aa9-86a0-94afaef6001e None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:11:59.846112 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e567edb-1b07-4aa9-86a0-94afaef6001e None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:11:59.846382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e567edb-1b07-4aa9-86a0-94afaef6001e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:00.005320 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-1e567edb-1b07-4aa9-86a0-94afaef6001e None None] MFA Rules not processed for user `e7ff3c53fa9b43afa9e77e467bd8dd58`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:00.006816 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-1e567edb-1b07-4aa9-86a0-94afaef6001e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:00.085279 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1194/2391] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:11:59 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:00.097311 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304b3e47-2ea2-40d7-9ac7-aed3ba62823b admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:00.099007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-304b3e47-2ea2-40d7-9ac7-aed3ba62823b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:00.166341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304b3e47-2ea2-40d7-9ac7-aed3ba62823b admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:00.171990 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c079406b-06f5-44f5-8ac9-69a65e038e24 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:00.172900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c079406b-06f5-44f5-8ac9-69a65e038e24 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:00.173381 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c079406b-06f5-44f5-8ac9-69a65e038e24 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:00.173799 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c079406b-06f5-44f5-8ac9-69a65e038e24 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:00.238384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c079406b-06f5-44f5-8ac9-69a65e038e24 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:00.239817 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c079406b-06f5-44f5-8ac9-69a65e038e24 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:00.248983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1198/2392] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:00 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:02.106820 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e567edb-1b07-4aa9-86a0-94afaef6001e None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:02.110864 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-1e567edb-1b07-4aa9-86a0-94afaef6001e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:02.126414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e567edb-1b07-4aa9-86a0-94afaef6001e None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:02.130168 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c37f952-b06b-4acb-9bf6-964627dc5153 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:02.130773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c37f952-b06b-4acb-9bf6-964627dc5153 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:02.131056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c37f952-b06b-4acb-9bf6-964627dc5153 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:02.131326 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c37f952-b06b-4acb-9bf6-964627dc5153 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:02.192904 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:02.230532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c37f952-b06b-4acb-9bf6-964627dc5153 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:02.231971 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c37f952-b06b-4acb-9bf6-964627dc5153 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:02.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1195/2393] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:02 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:03.674384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbd2d56-e8af-4473-92bb-e75976fddd0b None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:03.675235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbd2d56-e8af-4473-92bb-e75976fddd0b None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:03.675686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbd2d56-e8af-4473-92bb-e75976fddd0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:03.726893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-6fbd2d56-e8af-4473-92bb-e75976fddd0b None None] MFA Rules not processed for user `70cef7c271e046a3b49230057cc2d28f`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:03.728368 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-6fbd2d56-e8af-4473-92bb-e75976fddd0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:03.771967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:03.786002 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c37f952-b06b-4acb-9bf6-964627dc5153 service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:03.805237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c37f952-b06b-4acb-9bf6-964627dc5153 service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:03.811769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ebc25a-3d79-4ced-b3e2-e0df5061db0c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:03.818076 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1199/2394] 10.222.0.42 () {58 vars in 967 bytes} [Sun Sep 4 01:12:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:03.818484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ebc25a-3d79-4ced-b3e2-e0df5061db0c service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:03.818484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ebc25a-3d79-4ced-b3e2-e0df5061db0c service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:03.818484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ebc25a-3d79-4ced-b3e2-e0df5061db0c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:03.826921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbd2d56-e8af-4473-92bb-e75976fddd0b None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:03.827338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ebc25a-3d79-4ced-b3e2-e0df5061db0c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:03.828487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-6fbd2d56-e8af-4473-92bb-e75976fddd0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:03.828764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ebc25a-3d79-4ced-b3e2-e0df5061db0c service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:03.846654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1196/2395] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:03 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:03.922563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbd2d56-e8af-4473-92bb-e75976fddd0b None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:03.937275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68592f36-6531-4cb3-8918-6e9cf72ff524 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '123ab6ff796c4e599ce1d6f4095bd898', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:03.937275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68592f36-6531-4cb3-8918-6e9cf72ff524 service swift] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:03.937275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68592f36-6531-4cb3-8918-6e9cf72ff524 service swift] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:03.937275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68592f36-6531-4cb3-8918-6e9cf72ff524 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:03.940474 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68592f36-6531-4cb3-8918-6e9cf72ff524 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:03.940474 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68592f36-6531-4cb3-8918-6e9cf72ff524 service swift] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:03.947929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1200/2396] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 4 01:12:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 04 01:12:08.424939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ebc25a-3d79-4ced-b3e2-e0df5061db0c service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:08.489258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ebc25a-3d79-4ced-b3e2-e0df5061db0c service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:08.496635 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c0d51d-9989-4a44-ba15-53e9a87cb6df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:08.496635 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c0d51d-9989-4a44-ba15-53e9a87cb6df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:08.496635 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c0d51d-9989-4a44-ba15-53e9a87cb6df admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:08.496635 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c0d51d-9989-4a44-ba15-53e9a87cb6df admin admin] PATH_INFO: `/v3/users/5608e90dc7bf4ca89b4f1127a5f14088` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:08.498211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c0d51d-9989-4a44-ba15-53e9a87cb6df admin admin] RBAC: Authorizing `identity:delete_user(user_id=5608e90dc7bf4ca89b4f1127a5f14088)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:08.499099 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c0d51d-9989-4a44-ba15-53e9a87cb6df admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:08.712985 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-30c0d51d-9989-4a44-ba15-53e9a87cb6df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5608e90dc7bf4ca89b4f1127a5f14088'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:08.713738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-30c0d51d-9989-4a44-ba15-53e9a87cb6df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5608e90dc7bf4ca89b4f1127a5f14088'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:08.768819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-30c0d51d-9989-4a44-ba15-53e9a87cb6df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5608e90dc7bf4ca89b4f1127a5f14088'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:08.782073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-30c0d51d-9989-4a44-ba15-53e9a87cb6df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5608e90dc7bf4ca89b4f1127a5f14088'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:08.782727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-30c0d51d-9989-4a44-ba15-53e9a87cb6df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5608e90dc7bf4ca89b4f1127a5f14088'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:08.794719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1197/2397] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:08 2022] DELETE /identity/v3/users/5608e90dc7bf4ca89b4f1127a5f14088 => generated 0 bytes in 370 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:08.799889 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d271ad7-0032-4ae8-be95-cadb6bb3c875 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:08.800376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d271ad7-0032-4ae8-be95-cadb6bb3c875 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:08.800487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d271ad7-0032-4ae8-be95-cadb6bb3c875 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:08.820463 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9d271ad7-0032-4ae8-be95-cadb6bb3c875 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:08.821718 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9d271ad7-0032-4ae8-be95-cadb6bb3c875 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:08.885405 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1201/2398] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:08.895309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c0d51d-9989-4a44-ba15-53e9a87cb6df admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:08.896809 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-30c0d51d-9989-4a44-ba15-53e9a87cb6df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:08.938660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:08.956407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c0d51d-9989-4a44-ba15-53e9a87cb6df admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:08.960105 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed5dc9f-996b-4b85-99c1-460a4862d2d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:08.960891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed5dc9f-996b-4b85-99c1-460a4862d2d0 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:08.961195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed5dc9f-996b-4b85-99c1-460a4862d2d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:08.961465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed5dc9f-996b-4b85-99c1-460a4862d2d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:08.971040 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed5dc9f-996b-4b85-99c1-460a4862d2d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:08.972473 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed5dc9f-996b-4b85-99c1-460a4862d2d0 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:08.981746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1198/2399] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:09.467569 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d271ad7-0032-4ae8-be95-cadb6bb3c875 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:09.469407 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9d271ad7-0032-4ae8-be95-cadb6bb3c875 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:09.482671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d271ad7-0032-4ae8-be95-cadb6bb3c875 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:09.489102 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2818a6a-93bd-4432-aa25-5e92c35b1157 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:09.489797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2818a6a-93bd-4432-aa25-5e92c35b1157 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:09.490078 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2818a6a-93bd-4432-aa25-5e92c35b1157 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:09.490456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2818a6a-93bd-4432-aa25-5e92c35b1157 admin admin] PATH_INFO: `/v3/projects/4a9de8cefc614e71b43a8ccead0d671a` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:09.491804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2818a6a-93bd-4432-aa25-5e92c35b1157 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4a9de8cefc614e71b43a8ccead0d671a)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:09.504178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2818a6a-93bd-4432-aa25-5e92c35b1157 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:09.691142 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b2818a6a-93bd-4432-aa25-5e92c35b1157 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4a9de8cefc614e71b43a8ccead0d671a', 'request_id': 'req-b2818a6a-93bd-4432-aa25-5e92c35b1157'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:09.697130 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b2818a6a-93bd-4432-aa25-5e92c35b1157 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4a9de8cefc614e71b43a8ccead0d671a', 'request_id': 'req-b2818a6a-93bd-4432-aa25-5e92c35b1157'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:09.698349 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:09.704060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b2818a6a-93bd-4432-aa25-5e92c35b1157 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4a9de8cefc614e71b43a8ccead0d671a', 'request_id': 'req-b2818a6a-93bd-4432-aa25-5e92c35b1157'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:09.792175 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b2818a6a-93bd-4432-aa25-5e92c35b1157 admin admin] The token cache is being invalidate because project 4a9de8cefc614e71b43a8ccead0d671a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:12:09.792492 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b2818a6a-93bd-4432-aa25-5e92c35b1157 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:09.794189 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b2818a6a-93bd-4432-aa25-5e92c35b1157 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:09.799470 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1202/2400] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:12:09 2022] DELETE /identity/v3/projects/4a9de8cefc614e71b43a8ccead0d671a => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:09.800841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed5dc9f-996b-4b85-99c1-460a4862d2d0 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:09.810564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed5dc9f-996b-4b85-99c1-460a4862d2d0 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:09.877692 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7efd575-cbde-4bd1-9757-b09da1cc381d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:09.878340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7efd575-cbde-4bd1-9757-b09da1cc381d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:09.878626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7efd575-cbde-4bd1-9757-b09da1cc381d admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:09.878899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7efd575-cbde-4bd1-9757-b09da1cc381d admin admin] PATH_INFO: `/v3/users/baf4962efd214558997fac219f51950a` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:09.888891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7efd575-cbde-4bd1-9757-b09da1cc381d admin admin] RBAC: Authorizing `identity:delete_user(user_id=baf4962efd214558997fac219f51950a)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:09.888891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7efd575-cbde-4bd1-9757-b09da1cc381d admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:09.899079 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2818a6a-93bd-4432-aa25-5e92c35b1157 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:09.901590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b2818a6a-93bd-4432-aa25-5e92c35b1157 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:09.959326 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2818a6a-93bd-4432-aa25-5e92c35b1157 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:09.963526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2493f8-3ea9-41f6-a1d3-eca377969684 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:09.964088 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2493f8-3ea9-41f6-a1d3-eca377969684 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:09.964371 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2493f8-3ea9-41f6-a1d3-eca377969684 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:09.964647 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2493f8-3ea9-41f6-a1d3-eca377969684 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:09.966569 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-3b2493f8-3ea9-41f6-a1d3-eca377969684 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:09.979135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2493f8-3ea9-41f6-a1d3-eca377969684 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:09.980592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2493f8-3ea9-41f6-a1d3-eca377969684 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:09.991855 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1203/2401] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:10.057346 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c7efd575-cbde-4bd1-9757-b09da1cc381d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'baf4962efd214558997fac219f51950a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:10.058001 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c7efd575-cbde-4bd1-9757-b09da1cc381d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'baf4962efd214558997fac219f51950a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:10.157830 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c7efd575-cbde-4bd1-9757-b09da1cc381d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'baf4962efd214558997fac219f51950a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:10.172531 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c7efd575-cbde-4bd1-9757-b09da1cc381d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'baf4962efd214558997fac219f51950a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:10.173575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c7efd575-cbde-4bd1-9757-b09da1cc381d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'baf4962efd214558997fac219f51950a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:10.187657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1199/2402] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:09 2022] DELETE /identity/v3/users/baf4962efd214558997fac219f51950a => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:10.658250 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2493f8-3ea9-41f6-a1d3-eca377969684 service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:10.659791 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-3b2493f8-3ea9-41f6-a1d3-eca377969684 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:10.673360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2493f8-3ea9-41f6-a1d3-eca377969684 service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:10.747233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:10.747877 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:10.748166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:10.748443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42 admin admin] PATH_INFO: `/v3/projects/7b00179d801c40fd8d01bcb03cf3f14b` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:10.749900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b00179d801c40fd8d01bcb03cf3f14b)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:10.752084 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:10.868257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b00179d801c40fd8d01bcb03cf3f14b', 'request_id': 'req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:10.875048 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b00179d801c40fd8d01bcb03cf3f14b', 'request_id': 'req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:10.881568 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b00179d801c40fd8d01bcb03cf3f14b', 'request_id': 'req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:10.962489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42 admin admin] The token cache is being invalidate because project 7b00179d801c40fd8d01bcb03cf3f14b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:12:10.963101 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:10.963553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:10.965287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1204/2403] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:12:10 2022] DELETE /identity/v3/projects/7b00179d801c40fd8d01bcb03cf3f14b => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:10.969557 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7efd575-cbde-4bd1-9757-b09da1cc381d admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:10.972924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-c7efd575-cbde-4bd1-9757-b09da1cc381d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:10.984432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7efd575-cbde-4bd1-9757-b09da1cc381d admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:11.038899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:11.045151 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff55322d-0466-4fe1-85d9-d3faf489936d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:11.045799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff55322d-0466-4fe1-85d9-d3faf489936d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:11.046075 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff55322d-0466-4fe1-85d9-d3faf489936d admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:11.046957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff55322d-0466-4fe1-85d9-d3faf489936d admin admin] PATH_INFO: `/v3/users/e7ff3c53fa9b43afa9e77e467bd8dd58` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:11.048291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff55322d-0466-4fe1-85d9-d3faf489936d admin admin] RBAC: Authorizing `identity:delete_user(user_id=e7ff3c53fa9b43afa9e77e467bd8dd58)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:11.050254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff55322d-0466-4fe1-85d9-d3faf489936d admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:11.268293 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ff55322d-0466-4fe1-85d9-d3faf489936d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e7ff3c53fa9b43afa9e77e467bd8dd58'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:11.268941 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ff55322d-0466-4fe1-85d9-d3faf489936d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e7ff3c53fa9b43afa9e77e467bd8dd58'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:11.329127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ff55322d-0466-4fe1-85d9-d3faf489936d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e7ff3c53fa9b43afa9e77e467bd8dd58'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:11.358301 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ff55322d-0466-4fe1-85d9-d3faf489936d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e7ff3c53fa9b43afa9e77e467bd8dd58'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:11.360087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ff55322d-0466-4fe1-85d9-d3faf489936d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e7ff3c53fa9b43afa9e77e467bd8dd58'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:11.386455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1200/2404] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:10 2022] DELETE /identity/v3/users/e7ff3c53fa9b43afa9e77e467bd8dd58 => generated 0 bytes in 418 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:12.159707 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:12.167527 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:12.179533 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c50ac44-d8cb-4e8f-bf53-c986663a2f42 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:12.200465 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:12.251631 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7943d3-f9c4-46e0-91f1-eb70cdd6218f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:12.252276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7943d3-f9c4-46e0-91f1-eb70cdd6218f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:12.252553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7943d3-f9c4-46e0-91f1-eb70cdd6218f admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:12.252823 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7943d3-f9c4-46e0-91f1-eb70cdd6218f admin admin] PATH_INFO: `/v3/projects/b8b43148935f46118bf416336b1abdbe` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:12.254145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7943d3-f9c4-46e0-91f1-eb70cdd6218f admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8b43148935f46118bf416336b1abdbe)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:12.256140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7943d3-f9c4-46e0-91f1-eb70cdd6218f admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:12.393391 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-2a7943d3-f9c4-46e0-91f1-eb70cdd6218f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8b43148935f46118bf416336b1abdbe', 'request_id': 'req-2a7943d3-f9c4-46e0-91f1-eb70cdd6218f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:12.398973 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-2a7943d3-f9c4-46e0-91f1-eb70cdd6218f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8b43148935f46118bf416336b1abdbe', 'request_id': 'req-2a7943d3-f9c4-46e0-91f1-eb70cdd6218f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:12.405315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-2a7943d3-f9c4-46e0-91f1-eb70cdd6218f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8b43148935f46118bf416336b1abdbe', 'request_id': 'req-2a7943d3-f9c4-46e0-91f1-eb70cdd6218f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:12.489913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-2a7943d3-f9c4-46e0-91f1-eb70cdd6218f admin admin] The token cache is being invalidate because project b8b43148935f46118bf416336b1abdbe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:12:12.490264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-2a7943d3-f9c4-46e0-91f1-eb70cdd6218f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:12.492952 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-2a7943d3-f9c4-46e0-91f1-eb70cdd6218f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:12.500305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1205/2405] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:12:12 2022] DELETE /identity/v3/projects/b8b43148935f46118bf416336b1abdbe => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:12.504283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7ad179-8c1b-450a-8cda-37a48117ddd2 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:12.504283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7ad179-8c1b-450a-8cda-37a48117ddd2 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:12.504283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7ad179-8c1b-450a-8cda-37a48117ddd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:12.533325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-9d7ad179-8c1b-450a-8cda-37a48117ddd2 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:12.535292 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-9d7ad179-8c1b-450a-8cda-37a48117ddd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:12.626427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1201/2406] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:12.637118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2090d576-d839-42b2-8a3b-5d9c5abd0a9a None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:12.637537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2090d576-d839-42b2-8a3b-5d9c5abd0a9a None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:12.637926 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2090d576-d839-42b2-8a3b-5d9c5abd0a9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:12.664884 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-2090d576-d839-42b2-8a3b-5d9c5abd0a9a None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:12.666598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-2090d576-d839-42b2-8a3b-5d9c5abd0a9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:12.681729 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1206/2407] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:12.682892 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7ad179-8c1b-450a-8cda-37a48117ddd2 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:12.698288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7ad179-8c1b-450a-8cda-37a48117ddd2 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:12.721859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15674dd-0ba1-436c-94c5-f6f3930d3e90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:12.722498 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15674dd-0ba1-436c-94c5-f6f3930d3e90 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:12.728027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15674dd-0ba1-436c-94c5-f6f3930d3e90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:12.728027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15674dd-0ba1-436c-94c5-f6f3930d3e90 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:12.728027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15674dd-0ba1-436c-94c5-f6f3930d3e90 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:12:12.728027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15674dd-0ba1-436c-94c5-f6f3930d3e90 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:12.728027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15674dd-0ba1-436c-94c5-f6f3930d3e90 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:12.745137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1202/2408] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:12:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 01:12:12.746933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2090d576-d839-42b2-8a3b-5d9c5abd0a9a None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:12.763677 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2090d576-d839-42b2-8a3b-5d9c5abd0a9a None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:12.792910 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46ea834-995f-4782-8a90-9f85d9324215 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:12.792910 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46ea834-995f-4782-8a90-9f85d9324215 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:12.792910 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46ea834-995f-4782-8a90-9f85d9324215 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:12.792910 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46ea834-995f-4782-8a90-9f85d9324215 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:12.793421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46ea834-995f-4782-8a90-9f85d9324215 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:12.793896 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46ea834-995f-4782-8a90-9f85d9324215 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:12.906847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1207/2409] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:12:12 2022] POST /identity/v3/projects => generated 385 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:12:12.912200 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15674dd-0ba1-436c-94c5-f6f3930d3e90 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:12.941106 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15674dd-0ba1-436c-94c5-f6f3930d3e90 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:13.009519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cd4bdd-b410-439d-9e5b-dc58b7f93dfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:13.010146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cd4bdd-b410-439d-9e5b-dc58b7f93dfc admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:13.010410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cd4bdd-b410-439d-9e5b-dc58b7f93dfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:13.011208 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cd4bdd-b410-439d-9e5b-dc58b7f93dfc admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:13.011341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cd4bdd-b410-439d-9e5b-dc58b7f93dfc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:13.013447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cd4bdd-b410-439d-9e5b-dc58b7f93dfc admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:13.025324 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=87,keystone:DELETE=10,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:13.113920 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1203/2410] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:12:12 2022] POST /identity/v3/users => generated 293 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:13.118701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46ea834-995f-4782-8a90-9f85d9324215 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:13.130437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46ea834-995f-4782-8a90-9f85d9324215 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:13.137503 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3179e00e-5b75-4397-8b32-1458bc929882 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:13.138111 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3179e00e-5b75-4397-8b32-1458bc929882 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:13.138390 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3179e00e-5b75-4397-8b32-1458bc929882 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:13.138866 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3179e00e-5b75-4397-8b32-1458bc929882 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:13.139648 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3179e00e-5b75-4397-8b32-1458bc929882 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:13.142109 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3179e00e-5b75-4397-8b32-1458bc929882 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:13.153747 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1208/2411] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:13 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:13.158290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cd4bdd-b410-439d-9e5b-dc58b7f93dfc admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:13.170870 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cd4bdd-b410-439d-9e5b-dc58b7f93dfc admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:13.175609 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044167b7-0613-45cc-9599-688145a8d4fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:13.179988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044167b7-0613-45cc-9599-688145a8d4fd admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:13.179988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044167b7-0613-45cc-9599-688145a8d4fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:13.179988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044167b7-0613-45cc-9599-688145a8d4fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:13.179988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044167b7-0613-45cc-9599-688145a8d4fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:13.179988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044167b7-0613-45cc-9599-688145a8d4fd admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:13.195365 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3179e00e-5b75-4397-8b32-1458bc929882 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:13.204169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1204/2412] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:13 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:13.209453 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3179e00e-5b75-4397-8b32-1458bc929882 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:13.216139 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745af9e1-bff9-4f01-9e8f-2c33e49870ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:13.217321 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745af9e1-bff9-4f01-9e8f-2c33e49870ce admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:13.217760 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745af9e1-bff9-4f01-9e8f-2c33e49870ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:13.218845 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745af9e1-bff9-4f01-9e8f-2c33e49870ce admin admin] PATH_INFO: `/v3/projects/6cfa564295f4423ead002277f3c4b3c2/users/1cf8c7ec782c429a9773711ee04923b1/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:13.244342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745af9e1-bff9-4f01-9e8f-2c33e49870ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=6cfa564295f4423ead002277f3c4b3c2, user_id=1cf8c7ec782c429a9773711ee04923b1, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:13.247485 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745af9e1-bff9-4f01-9e8f-2c33e49870ce admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:13.330320 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1209/2413] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:13 2022] PUT /identity/v3/projects/6cfa564295f4423ead002277f3c4b3c2/users/1cf8c7ec782c429a9773711ee04923b1/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:12:13.340993 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46f36b1-529b-4305-b376-73f63f8cdecb None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:13.341328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46f36b1-529b-4305-b376-73f63f8cdecb None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:13.356330 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46f36b1-529b-4305-b376-73f63f8cdecb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:13.373417 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-f46f36b1-529b-4305-b376-73f63f8cdecb None None] MFA Rules not processed for user `1cf8c7ec782c429a9773711ee04923b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:13.374539 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f46f36b1-529b-4305-b376-73f63f8cdecb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:13.457780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1205/2414] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:13 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:13.469741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745af9e1-bff9-4f01-9e8f-2c33e49870ce admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:13.474716 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-745af9e1-bff9-4f01-9e8f-2c33e49870ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:13.536769 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:13.541930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745af9e1-bff9-4f01-9e8f-2c33e49870ce admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:13.545677 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fd8b16-9329-4e5c-8c6a-ee9442b0b819 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:13.546530 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fd8b16-9329-4e5c-8c6a-ee9442b0b819 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:13.546951 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fd8b16-9329-4e5c-8c6a-ee9442b0b819 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:13.547471 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fd8b16-9329-4e5c-8c6a-ee9442b0b819 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:13.555868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fd8b16-9329-4e5c-8c6a-ee9442b0b819 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:13.557399 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fd8b16-9329-4e5c-8c6a-ee9442b0b819 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:13.566950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1210/2415] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:13 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:14.546924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46f36b1-529b-4305-b376-73f63f8cdecb None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:14.548826 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f46f36b1-529b-4305-b376-73f63f8cdecb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:14.641714 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46f36b1-529b-4305-b376-73f63f8cdecb None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:14.648097 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb67e62c-a0f5-4a9d-9164-fc1595a97506 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b29f444d56ec49308f7fef16f4d1eea5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:14.648801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb67e62c-a0f5-4a9d-9164-fc1595a97506 service placement] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:14.649184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb67e62c-a0f5-4a9d-9164-fc1595a97506 service placement] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:14.649529 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb67e62c-a0f5-4a9d-9164-fc1595a97506 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:14.652247 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-eb67e62c-a0f5-4a9d-9164-fc1595a97506 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:14.666620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb67e62c-a0f5-4a9d-9164-fc1595a97506 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:14.668321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb67e62c-a0f5-4a9d-9164-fc1595a97506 service placement] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:14.680085 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1206/2416] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:14 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:14.695067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fd8b16-9329-4e5c-8c6a-ee9442b0b819 service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:14.697694 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-f7fd8b16-9329-4e5c-8c6a-ee9442b0b819 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:14.781897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fd8b16-9329-4e5c-8c6a-ee9442b0b819 service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:14.786646 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4a98f5-184d-48f7-8d3c-b86c75b96594 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:14.787267 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4a98f5-184d-48f7-8d3c-b86c75b96594 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:14.787542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4a98f5-184d-48f7-8d3c-b86c75b96594 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:14.788971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4a98f5-184d-48f7-8d3c-b86c75b96594 admin admin] PATH_INFO: `/v3/users/44993cd93360442bb0afb28778ee83bc` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:14.788971 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4a98f5-184d-48f7-8d3c-b86c75b96594 admin admin] RBAC: Authorizing `identity:delete_user(user_id=44993cd93360442bb0afb28778ee83bc)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:14.790864 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4a98f5-184d-48f7-8d3c-b86c75b96594 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:15.006481 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3c4a98f5-184d-48f7-8d3c-b86c75b96594 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44993cd93360442bb0afb28778ee83bc'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:15.095244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3c4a98f5-184d-48f7-8d3c-b86c75b96594 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44993cd93360442bb0afb28778ee83bc'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:15.110204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3c4a98f5-184d-48f7-8d3c-b86c75b96594 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44993cd93360442bb0afb28778ee83bc'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:15.111375 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3c4a98f5-184d-48f7-8d3c-b86c75b96594 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44993cd93360442bb0afb28778ee83bc'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:15.119807 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-3c4a98f5-184d-48f7-8d3c-b86c75b96594 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44993cd93360442bb0afb28778ee83bc'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:15.126177 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1211/2417] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:14 2022] DELETE /identity/v3/users/44993cd93360442bb0afb28778ee83bc => generated 0 bytes in 432 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:15.393512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb67e62c-a0f5-4a9d-9164-fc1595a97506 service placement] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:15.405516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb67e62c-a0f5-4a9d-9164-fc1595a97506 service placement] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:15.492317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e50d8eb-3423-44ae-baaf-20ad387b8f82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:15.492953 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e50d8eb-3423-44ae-baaf-20ad387b8f82 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:15.493270 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e50d8eb-3423-44ae-baaf-20ad387b8f82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:15.493550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e50d8eb-3423-44ae-baaf-20ad387b8f82 admin admin] PATH_INFO: `/v3/projects/e7dca68418a14d7da9aa63ac25f202e8` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:15.494787 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e50d8eb-3423-44ae-baaf-20ad387b8f82 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e7dca68418a14d7da9aa63ac25f202e8)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:15.496807 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e50d8eb-3423-44ae-baaf-20ad387b8f82 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:15.500543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:15.656765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8e50d8eb-3423-44ae-baaf-20ad387b8f82 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e7dca68418a14d7da9aa63ac25f202e8', 'request_id': 'req-8e50d8eb-3423-44ae-baaf-20ad387b8f82'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:15.663989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8e50d8eb-3423-44ae-baaf-20ad387b8f82 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e7dca68418a14d7da9aa63ac25f202e8', 'request_id': 'req-8e50d8eb-3423-44ae-baaf-20ad387b8f82'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:15.762693 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8e50d8eb-3423-44ae-baaf-20ad387b8f82 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e7dca68418a14d7da9aa63ac25f202e8', 'request_id': 'req-8e50d8eb-3423-44ae-baaf-20ad387b8f82'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:15.776727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8e50d8eb-3423-44ae-baaf-20ad387b8f82 admin admin] The token cache is being invalidate because project e7dca68418a14d7da9aa63ac25f202e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:12:15.777120 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8e50d8eb-3423-44ae-baaf-20ad387b8f82 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:15.778731 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8e50d8eb-3423-44ae-baaf-20ad387b8f82 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:15.781380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1207/2418] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:12:15 2022] DELETE /identity/v3/projects/e7dca68418a14d7da9aa63ac25f202e8 => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:15.786579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a526cca-2d0a-42e7-910b-c8a6b1a6bd76 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:15.787040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a526cca-2d0a-42e7-910b-c8a6b1a6bd76 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:15.787126 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a526cca-2d0a-42e7-910b-c8a6b1a6bd76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:15.820150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9a526cca-2d0a-42e7-910b-c8a6b1a6bd76 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:15.820150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9a526cca-2d0a-42e7-910b-c8a6b1a6bd76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:15.900130 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1212/2419] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:15.915887 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c937c5-e778-4098-9651-cf1332cbfbef None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:15.916166 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c937c5-e778-4098-9651-cf1332cbfbef None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:15.916542 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c937c5-e778-4098-9651-cf1332cbfbef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:15.949620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-83c937c5-e778-4098-9651-cf1332cbfbef None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:15.951420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-83c937c5-e778-4098-9651-cf1332cbfbef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:15.962871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1208/2420] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:15.967651 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a526cca-2d0a-42e7-910b-c8a6b1a6bd76 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:15.979026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a526cca-2d0a-42e7-910b-c8a6b1a6bd76 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:15.984076 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9cce83-0eb1-4c60-8965-f5a8b5e72287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:15.984696 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9cce83-0eb1-4c60-8965-f5a8b5e72287 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:15.984979 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9cce83-0eb1-4c60-8965-f5a8b5e72287 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:15.992361 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9cce83-0eb1-4c60-8965-f5a8b5e72287 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:15.992361 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9cce83-0eb1-4c60-8965-f5a8b5e72287 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:12:15.992361 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9cce83-0eb1-4c60-8965-f5a8b5e72287 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:15.992361 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9cce83-0eb1-4c60-8965-f5a8b5e72287 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:15.992361 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:16.001837 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1213/2421] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:12:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:12:16.006304 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c937c5-e778-4098-9651-cf1332cbfbef None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:16.016720 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c937c5-e778-4098-9651-cf1332cbfbef None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:16.021441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0587f9e-a80c-4583-8541-662b93e691ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:16.022045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0587f9e-a80c-4583-8541-662b93e691ea admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:16.022323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0587f9e-a80c-4583-8541-662b93e691ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:16.022592 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0587f9e-a80c-4583-8541-662b93e691ea admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:16.023244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0587f9e-a80c-4583-8541-662b93e691ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:16.025182 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0587f9e-a80c-4583-8541-662b93e691ea admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:16.166258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1209/2422] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:12:16 2022] POST /identity/v3/projects => generated 405 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:16.170570 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9cce83-0eb1-4c60-8965-f5a8b5e72287 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:16.181388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9cce83-0eb1-4c60-8965-f5a8b5e72287 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:16.245453 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa64f08-7b0a-4bcb-a820-d3b14b983549 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:16.246132 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa64f08-7b0a-4bcb-a820-d3b14b983549 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:16.246405 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa64f08-7b0a-4bcb-a820-d3b14b983549 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:16.246681 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa64f08-7b0a-4bcb-a820-d3b14b983549 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:16.247329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa64f08-7b0a-4bcb-a820-d3b14b983549 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:16.253278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa64f08-7b0a-4bcb-a820-d3b14b983549 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:16.375239 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1214/2423] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:12:16 2022] POST /identity/v3/users => generated 303 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:16.379433 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0587f9e-a80c-4583-8541-662b93e691ea admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:16.390473 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0587f9e-a80c-4583-8541-662b93e691ea admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:16.395082 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c3c189-cc48-49c4-bd6b-05ef6d3272ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:16.395724 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c3c189-cc48-49c4-bd6b-05ef6d3272ae admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:16.396071 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c3c189-cc48-49c4-bd6b-05ef6d3272ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:16.396347 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c3c189-cc48-49c4-bd6b-05ef6d3272ae admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:16.397300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c3c189-cc48-49c4-bd6b-05ef6d3272ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:16.398902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c3c189-cc48-49c4-bd6b-05ef6d3272ae admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:16.411043 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1210/2424] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:16 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:16.416508 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa64f08-7b0a-4bcb-a820-d3b14b983549 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:16.428046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa64f08-7b0a-4bcb-a820-d3b14b983549 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:16.434185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622c832b-d1e4-4b08-bf8a-e4cccbbbc236 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:16.437248 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622c832b-d1e4-4b08-bf8a-e4cccbbbc236 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:16.437248 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622c832b-d1e4-4b08-bf8a-e4cccbbbc236 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:16.437248 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622c832b-d1e4-4b08-bf8a-e4cccbbbc236 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:16.437248 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622c832b-d1e4-4b08-bf8a-e4cccbbbc236 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:16.437696 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622c832b-d1e4-4b08-bf8a-e4cccbbbc236 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:16.449586 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1215/2425] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:16 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:16.454136 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c3c189-cc48-49c4-bd6b-05ef6d3272ae admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:16.464974 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c3c189-cc48-49c4-bd6b-05ef6d3272ae admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:16.471215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73051464-f578-4bbe-bcc1-7756b576ca91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:16.471336 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73051464-f578-4bbe-bcc1-7756b576ca91 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:16.471595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73051464-f578-4bbe-bcc1-7756b576ca91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:16.471880 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73051464-f578-4bbe-bcc1-7756b576ca91 admin admin] PATH_INFO: `/v3/projects/5c147b2ed0724f149841d070b16f749d/users/882d80a015ad4cbebad53282bb7d1491/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:16.488720 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73051464-f578-4bbe-bcc1-7756b576ca91 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c147b2ed0724f149841d070b16f749d, user_id=882d80a015ad4cbebad53282bb7d1491, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:16.491644 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73051464-f578-4bbe-bcc1-7756b576ca91 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:16.585628 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1211/2426] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:16 2022] PUT /identity/v3/projects/5c147b2ed0724f149841d070b16f749d/users/882d80a015ad4cbebad53282bb7d1491/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:16.596915 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82655aa-9ae8-47af-ae8b-f24bde0e23ea None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:16.597782 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82655aa-9ae8-47af-ae8b-f24bde0e23ea None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:16.598146 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82655aa-9ae8-47af-ae8b-f24bde0e23ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:16.633549 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-a82655aa-9ae8-47af-ae8b-f24bde0e23ea None None] MFA Rules not processed for user `882d80a015ad4cbebad53282bb7d1491`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:16.634908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a82655aa-9ae8-47af-ae8b-f24bde0e23ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:16.705104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1216/2427] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:16 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:16.707671 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73051464-f578-4bbe-bcc1-7756b576ca91 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:16.771767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73051464-f578-4bbe-bcc1-7756b576ca91 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:16.776551 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816e7604-3323-4acf-9c0d-aea475f80543 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:16.777534 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816e7604-3323-4acf-9c0d-aea475f80543 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:16.777534 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816e7604-3323-4acf-9c0d-aea475f80543 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:16.777787 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816e7604-3323-4acf-9c0d-aea475f80543 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:16.778432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816e7604-3323-4acf-9c0d-aea475f80543 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:16.780348 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816e7604-3323-4acf-9c0d-aea475f80543 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:16.876775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1212/2428] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:12:16 2022] POST /identity/v3/projects => generated 405 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:16.877839 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816e7604-3323-4acf-9c0d-aea475f80543 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:16.877981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82655aa-9ae8-47af-ae8b-f24bde0e23ea None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:16.879526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-816e7604-3323-4acf-9c0d-aea475f80543 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:16.891543 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82655aa-9ae8-47af-ae8b-f24bde0e23ea None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:16.899923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:16.950271 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816e7604-3323-4acf-9c0d-aea475f80543 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:16.955533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb6984f-52b5-4514-8520-cd0dbc111fe1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:16.956212 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb6984f-52b5-4514-8520-cd0dbc111fe1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:16.956494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb6984f-52b5-4514-8520-cd0dbc111fe1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:16.956765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb6984f-52b5-4514-8520-cd0dbc111fe1 admin admin] PATH_INFO: `/v3/users/1cf8c7ec782c429a9773711ee04923b1` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:16.957961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb6984f-52b5-4514-8520-cd0dbc111fe1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1cf8c7ec782c429a9773711ee04923b1)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:16.959896 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb6984f-52b5-4514-8520-cd0dbc111fe1 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:16.972038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cc3584-7440-435d-890d-23dcca024c5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:16.972649 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cc3584-7440-435d-890d-23dcca024c5d admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:16.972925 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cc3584-7440-435d-890d-23dcca024c5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:16.973237 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cc3584-7440-435d-890d-23dcca024c5d admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:16.973867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cc3584-7440-435d-890d-23dcca024c5d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:16.975782 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cc3584-7440-435d-890d-23dcca024c5d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:16.986159 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:17.071358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1217/2429] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:12:16 2022] POST /identity/v3/users => generated 309 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:12:17.075274 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cc3584-7440-435d-890d-23dcca024c5d admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:17.086438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cc3584-7440-435d-890d-23dcca024c5d admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:17.091279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b121e53e-8aa6-4359-ba9e-02400179a9bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:17.091938 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b121e53e-8aa6-4359-ba9e-02400179a9bd admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:17.092222 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b121e53e-8aa6-4359-ba9e-02400179a9bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:17.092496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b121e53e-8aa6-4359-ba9e-02400179a9bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:17.093180 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b121e53e-8aa6-4359-ba9e-02400179a9bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:17.094784 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b121e53e-8aa6-4359-ba9e-02400179a9bd admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:17.104813 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1218/2430] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:17 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:12:17.108968 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b121e53e-8aa6-4359-ba9e-02400179a9bd admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:17.118599 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b121e53e-8aa6-4359-ba9e-02400179a9bd admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:17.123368 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cbdc79-4fca-4d6e-a85e-f8662edb747f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:17.123992 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cbdc79-4fca-4d6e-a85e-f8662edb747f admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:17.124767 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cbdc79-4fca-4d6e-a85e-f8662edb747f admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:17.125047 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cbdc79-4fca-4d6e-a85e-f8662edb747f admin admin] PATH_INFO: `/v3/projects/82f9366e31bc45a6b94fafa49bdec9c9/users/f3eb2cb88ab94536b7e4b2dda60065bb/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:17.139446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cbdc79-4fca-4d6e-a85e-f8662edb747f admin admin] RBAC: Authorizing `identity:create_grant(project_id=82f9366e31bc45a6b94fafa49bdec9c9, user_id=f3eb2cb88ab94536b7e4b2dda60065bb, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:17.143509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cbdc79-4fca-4d6e-a85e-f8662edb747f admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:17.237101 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1219/2431] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:17 2022] PUT /identity/v3/projects/82f9366e31bc45a6b94fafa49bdec9c9/users/f3eb2cb88ab94536b7e4b2dda60065bb/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:12:17.241279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cbdc79-4fca-4d6e-a85e-f8662edb747f admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:17.246295 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5eb6984f-52b5-4514-8520-cd0dbc111fe1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1cf8c7ec782c429a9773711ee04923b1'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:17.246946 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5eb6984f-52b5-4514-8520-cd0dbc111fe1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1cf8c7ec782c429a9773711ee04923b1'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:17.304220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cbdc79-4fca-4d6e-a85e-f8662edb747f admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:17.308847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396f3388-d503-49d5-92e6-7eb891645fb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:17.309552 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f3388-d503-49d5-92e6-7eb891645fb2 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:17.309828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f3388-d503-49d5-92e6-7eb891645fb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:17.310100 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f3388-d503-49d5-92e6-7eb891645fb2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:17.310767 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396f3388-d503-49d5-92e6-7eb891645fb2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:17.312510 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396f3388-d503-49d5-92e6-7eb891645fb2 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:17.322059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1220/2432] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:17 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:12:17.326184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396f3388-d503-49d5-92e6-7eb891645fb2 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:17.327912 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-396f3388-d503-49d5-92e6-7eb891645fb2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:17.340212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396f3388-d503-49d5-92e6-7eb891645fb2 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:17.343544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5eb6984f-52b5-4514-8520-cd0dbc111fe1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1cf8c7ec782c429a9773711ee04923b1'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:17.347486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c51c94-9ed9-461f-a4e8-28b13b26fba6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:17.348480 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c51c94-9ed9-461f-a4e8-28b13b26fba6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:17.348933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c51c94-9ed9-461f-a4e8-28b13b26fba6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:17.349406 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c51c94-9ed9-461f-a4e8-28b13b26fba6 admin admin] PATH_INFO: `/v3/projects/82f9366e31bc45a6b94fafa49bdec9c9/users/f3eb2cb88ab94536b7e4b2dda60065bb/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:17.352393 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c51c94-9ed9-461f-a4e8-28b13b26fba6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=82f9366e31bc45a6b94fafa49bdec9c9, user_id=f3eb2cb88ab94536b7e4b2dda60065bb, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:17.357092 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c51c94-9ed9-461f-a4e8-28b13b26fba6 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:17.363535 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5eb6984f-52b5-4514-8520-cd0dbc111fe1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1cf8c7ec782c429a9773711ee04923b1'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:17.364623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5eb6984f-52b5-4514-8520-cd0dbc111fe1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1cf8c7ec782c429a9773711ee04923b1'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:17.374089 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1221/2433] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:17 2022] PUT /identity/v3/projects/82f9366e31bc45a6b94fafa49bdec9c9/users/f3eb2cb88ab94536b7e4b2dda60065bb/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:12:17.385176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce628a2f-4308-40f9-a1c5-014dfff2929e None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:17.385789 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce628a2f-4308-40f9-a1c5-014dfff2929e None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:17.386238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce628a2f-4308-40f9-a1c5-014dfff2929e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:17.389397 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1213/2434] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:16 2022] DELETE /identity/v3/users/1cf8c7ec782c429a9773711ee04923b1 => generated 0 bytes in 512 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:17.395907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dc92de-f215-4692-9a71-44dce591e8b6 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:17.396358 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dc92de-f215-4692-9a71-44dce591e8b6 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:17.396798 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dc92de-f215-4692-9a71-44dce591e8b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:17.418284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-83dc92de-f215-4692-9a71-44dce591e8b6 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:17.419492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-83dc92de-f215-4692-9a71-44dce591e8b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:17.420845 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-ce628a2f-4308-40f9-a1c5-014dfff2929e None None] MFA Rules not processed for user `f3eb2cb88ab94536b7e4b2dda60065bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:17.423104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ce628a2f-4308-40f9-a1c5-014dfff2929e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:17.484243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1214/2435] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:17.495564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dc92de-f215-4692-9a71-44dce591e8b6 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:17.497066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-83dc92de-f215-4692-9a71-44dce591e8b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:17.498383 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1222/2436] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:17 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:12:17.509423 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce628a2f-4308-40f9-a1c5-014dfff2929e None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:17.511027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ce628a2f-4308-40f9-a1c5-014dfff2929e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:17.533033 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:17.542410 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:17.557929 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dc92de-f215-4692-9a71-44dce591e8b6 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:17.558143 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0f1612-9cd3-4072-92d0-a0df5044a9ee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:17.558754 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0f1612-9cd3-4072-92d0-a0df5044a9ee service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:17.559018 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0f1612-9cd3-4072-92d0-a0df5044a9ee service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:17.559285 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0f1612-9cd3-4072-92d0-a0df5044a9ee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:17.568869 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0f1612-9cd3-4072-92d0-a0df5044a9ee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:17.570093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce628a2f-4308-40f9-a1c5-014dfff2929e None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:17.570322 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0f1612-9cd3-4072-92d0-a0df5044a9ee service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:17.573753 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f453e297-a1a3-4aa1-94cf-0d5982a7b237 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:17.574355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f453e297-a1a3-4aa1-94cf-0d5982a7b237 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:17.574658 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f453e297-a1a3-4aa1-94cf-0d5982a7b237 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:17.574930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f453e297-a1a3-4aa1-94cf-0d5982a7b237 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:17.580085 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1215/2437] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:17.584487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f453e297-a1a3-4aa1-94cf-0d5982a7b237 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:17.586016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f453e297-a1a3-4aa1-94cf-0d5982a7b237 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:17.596297 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1223/2438] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:17 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:12:17.772876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0f1612-9cd3-4072-92d0-a0df5044a9ee service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:17.783047 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0f1612-9cd3-4072-92d0-a0df5044a9ee service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:17.788231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8298de-8556-4696-baab-d2e5faf6a989 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:17.788850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8298de-8556-4696-baab-d2e5faf6a989 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:17.789181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8298de-8556-4696-baab-d2e5faf6a989 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:17.789463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8298de-8556-4696-baab-d2e5faf6a989 admin admin] PATH_INFO: `/v3/users/882d80a015ad4cbebad53282bb7d1491` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:17.790584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8298de-8556-4696-baab-d2e5faf6a989 admin admin] RBAC: Authorizing `identity:delete_user(user_id=882d80a015ad4cbebad53282bb7d1491)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:17.792488 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8298de-8556-4696-baab-d2e5faf6a989 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:17.970914 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-df8298de-8556-4696-baab-d2e5faf6a989 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '882d80a015ad4cbebad53282bb7d1491'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:17.971575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-df8298de-8556-4696-baab-d2e5faf6a989 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '882d80a015ad4cbebad53282bb7d1491'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:17.993363 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f453e297-a1a3-4aa1-94cf-0d5982a7b237 service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:17.994805 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-f453e297-a1a3-4aa1-94cf-0d5982a7b237 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:18.009661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f453e297-a1a3-4aa1-94cf-0d5982a7b237 service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:18.014287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb697251-45b3-4b6e-9b74-62575a2bf6e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:18.015485 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb697251-45b3-4b6e-9b74-62575a2bf6e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:18.015907 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb697251-45b3-4b6e-9b74-62575a2bf6e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:18.018755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb697251-45b3-4b6e-9b74-62575a2bf6e0 admin admin] PATH_INFO: `/v3/projects/6cfa564295f4423ead002277f3c4b3c2` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:18.018755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb697251-45b3-4b6e-9b74-62575a2bf6e0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6cfa564295f4423ead002277f3c4b3c2)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:18.019938 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb697251-45b3-4b6e-9b74-62575a2bf6e0 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:18.065838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-df8298de-8556-4696-baab-d2e5faf6a989 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '882d80a015ad4cbebad53282bb7d1491'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:18.080473 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-df8298de-8556-4696-baab-d2e5faf6a989 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '882d80a015ad4cbebad53282bb7d1491'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:18.081204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-df8298de-8556-4696-baab-d2e5faf6a989 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '882d80a015ad4cbebad53282bb7d1491'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:18.094150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1216/2439] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:17 2022] DELETE /identity/v3/users/882d80a015ad4cbebad53282bb7d1491 => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:18.098536 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59bdb96-ca65-445a-9484-8ab74a5a94f5 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:18.098935 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59bdb96-ca65-445a-9484-8ab74a5a94f5 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:18.099281 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59bdb96-ca65-445a-9484-8ab74a5a94f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:18.119775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-f59bdb96-ca65-445a-9484-8ab74a5a94f5 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:18.120982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f59bdb96-ca65-445a-9484-8ab74a5a94f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:18.143220 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bb697251-45b3-4b6e-9b74-62575a2bf6e0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6cfa564295f4423ead002277f3c4b3c2', 'request_id': 'req-bb697251-45b3-4b6e-9b74-62575a2bf6e0'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:18.149032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bb697251-45b3-4b6e-9b74-62575a2bf6e0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6cfa564295f4423ead002277f3c4b3c2', 'request_id': 'req-bb697251-45b3-4b6e-9b74-62575a2bf6e0'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:18.155902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bb697251-45b3-4b6e-9b74-62575a2bf6e0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6cfa564295f4423ead002277f3c4b3c2', 'request_id': 'req-bb697251-45b3-4b6e-9b74-62575a2bf6e0'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:18.182124 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1217/2440] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:18.192523 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59bdb96-ca65-445a-9484-8ab74a5a94f5 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:18.193909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f59bdb96-ca65-445a-9484-8ab74a5a94f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:18.231135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bb697251-45b3-4b6e-9b74-62575a2bf6e0 admin admin] The token cache is being invalidate because project 6cfa564295f4423ead002277f3c4b3c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:12:18.231423 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bb697251-45b3-4b6e-9b74-62575a2bf6e0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:18.232048 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bb697251-45b3-4b6e-9b74-62575a2bf6e0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:18.234501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1224/2441] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:12:17 2022] DELETE /identity/v3/projects/6cfa564295f4423ead002277f3c4b3c2 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:12:18.250461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:18.254708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59bdb96-ca65-445a-9484-8ab74a5a94f5 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:18.311179 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef957fe-a10c-4261-b2ff-93d2ee78f46c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:18.312579 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef957fe-a10c-4261-b2ff-93d2ee78f46c service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:18.312998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef957fe-a10c-4261-b2ff-93d2ee78f46c service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:18.313322 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef957fe-a10c-4261-b2ff-93d2ee78f46c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:18.315296 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-1ef957fe-a10c-4261-b2ff-93d2ee78f46c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:18.386865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef957fe-a10c-4261-b2ff-93d2ee78f46c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:18.388573 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef957fe-a10c-4261-b2ff-93d2ee78f46c service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:18.399196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1218/2442] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:12:18.787352 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb697251-45b3-4b6e-9b74-62575a2bf6e0 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:18.790869 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-bb697251-45b3-4b6e-9b74-62575a2bf6e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:18.802619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb697251-45b3-4b6e-9b74-62575a2bf6e0 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:18.807541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f499299a-4189-4e47-a926-0fd92e81e646 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:18.808176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f499299a-4189-4e47-a926-0fd92e81e646 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:18.808829 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f499299a-4189-4e47-a926-0fd92e81e646 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:18.808829 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f499299a-4189-4e47-a926-0fd92e81e646 admin admin] PATH_INFO: `/v3/projects/5c147b2ed0724f149841d070b16f749d` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:18.809854 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f499299a-4189-4e47-a926-0fd92e81e646 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c147b2ed0724f149841d070b16f749d)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:18.812344 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f499299a-4189-4e47-a926-0fd92e81e646 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:18.912257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f499299a-4189-4e47-a926-0fd92e81e646 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c147b2ed0724f149841d070b16f749d', 'request_id': 'req-f499299a-4189-4e47-a926-0fd92e81e646'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:18.917927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f499299a-4189-4e47-a926-0fd92e81e646 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c147b2ed0724f149841d070b16f749d', 'request_id': 'req-f499299a-4189-4e47-a926-0fd92e81e646'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:18.924706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f499299a-4189-4e47-a926-0fd92e81e646 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c147b2ed0724f149841d070b16f749d', 'request_id': 'req-f499299a-4189-4e47-a926-0fd92e81e646'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:18.979559 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f499299a-4189-4e47-a926-0fd92e81e646 admin admin] The token cache is being invalidate because project 5c147b2ed0724f149841d070b16f749d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:12:18.979850 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f499299a-4189-4e47-a926-0fd92e81e646 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:18.980479 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f499299a-4189-4e47-a926-0fd92e81e646 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:18.982007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1225/2443] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:12:18 2022] DELETE /identity/v3/projects/5c147b2ed0724f149841d070b16f749d => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:18.986118 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef957fe-a10c-4261-b2ff-93d2ee78f46c service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:18.996059 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef957fe-a10c-4261-b2ff-93d2ee78f46c service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:19.051619 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf3f125-61c0-47f0-aa35-e7c7406d3a9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:19.052243 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf3f125-61c0-47f0-aa35-e7c7406d3a9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:19.052521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf3f125-61c0-47f0-aa35-e7c7406d3a9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:19.052791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf3f125-61c0-47f0-aa35-e7c7406d3a9c admin admin] PATH_INFO: `/v3/users/f3eb2cb88ab94536b7e4b2dda60065bb` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:19.053937 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf3f125-61c0-47f0-aa35-e7c7406d3a9c admin admin] RBAC: Authorizing `identity:delete_user(user_id=f3eb2cb88ab94536b7e4b2dda60065bb)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:19.055823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf3f125-61c0-47f0-aa35-e7c7406d3a9c admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:19.228788 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-daf3f125-61c0-47f0-aa35-e7c7406d3a9c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f3eb2cb88ab94536b7e4b2dda60065bb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:19.229543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-daf3f125-61c0-47f0-aa35-e7c7406d3a9c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f3eb2cb88ab94536b7e4b2dda60065bb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:19.311183 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-daf3f125-61c0-47f0-aa35-e7c7406d3a9c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f3eb2cb88ab94536b7e4b2dda60065bb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:19.314170 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:19.324744 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-daf3f125-61c0-47f0-aa35-e7c7406d3a9c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f3eb2cb88ab94536b7e4b2dda60065bb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:19.325406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-daf3f125-61c0-47f0-aa35-e7c7406d3a9c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f3eb2cb88ab94536b7e4b2dda60065bb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:19.337464 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1219/2444] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:18 2022] DELETE /identity/v3/users/f3eb2cb88ab94536b7e4b2dda60065bb => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:19.754417 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f499299a-4189-4e47-a926-0fd92e81e646 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:19.755933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-f499299a-4189-4e47-a926-0fd92e81e646 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:19.769915 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f499299a-4189-4e47-a926-0fd92e81e646 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:19.807061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:19.833629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4352c3e6-e24f-40bd-820f-887f07208742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:19.834304 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4352c3e6-e24f-40bd-820f-887f07208742 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:19.834586 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4352c3e6-e24f-40bd-820f-887f07208742 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:19.834863 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4352c3e6-e24f-40bd-820f-887f07208742 admin admin] PATH_INFO: `/v3/projects/82f9366e31bc45a6b94fafa49bdec9c9` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:19.836087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4352c3e6-e24f-40bd-820f-887f07208742 admin admin] RBAC: Authorizing `identity:delete_project(project_id=82f9366e31bc45a6b94fafa49bdec9c9)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:19.838029 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4352c3e6-e24f-40bd-820f-887f07208742 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:19.945481 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4352c3e6-e24f-40bd-820f-887f07208742 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '82f9366e31bc45a6b94fafa49bdec9c9', 'request_id': 'req-4352c3e6-e24f-40bd-820f-887f07208742'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:19.950839 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4352c3e6-e24f-40bd-820f-887f07208742 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '82f9366e31bc45a6b94fafa49bdec9c9', 'request_id': 'req-4352c3e6-e24f-40bd-820f-887f07208742'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:19.957684 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4352c3e6-e24f-40bd-820f-887f07208742 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '82f9366e31bc45a6b94fafa49bdec9c9', 'request_id': 'req-4352c3e6-e24f-40bd-820f-887f07208742'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:20.042797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4352c3e6-e24f-40bd-820f-887f07208742 admin admin] The token cache is being invalidate because project 82f9366e31bc45a6b94fafa49bdec9c9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:12:20.043001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4352c3e6-e24f-40bd-820f-887f07208742 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:20.043611 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4352c3e6-e24f-40bd-820f-887f07208742 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:20.045134 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1226/2445] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:12:19 2022] DELETE /identity/v3/projects/82f9366e31bc45a6b94fafa49bdec9c9 => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:20.050292 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02030e2e-3400-4635-b36d-13b9d05fbccc None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:20.050798 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02030e2e-3400-4635-b36d-13b9d05fbccc None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:20.051030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02030e2e-3400-4635-b36d-13b9d05fbccc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:20.073852 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-02030e2e-3400-4635-b36d-13b9d05fbccc None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:20.075606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-02030e2e-3400-4635-b36d-13b9d05fbccc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:20.136355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1220/2446] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:20.146409 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e4b9e-b157-444e-94c4-93e50736d41f None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:20.146696 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e4b9e-b157-444e-94c4-93e50736d41f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:20.146973 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e4b9e-b157-444e-94c4-93e50736d41f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:20.166597 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-666e4b9e-b157-444e-94c4-93e50736d41f None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:20.167798 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-666e4b9e-b157-444e-94c4-93e50736d41f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:20.175870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1227/2447] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:20.180199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02030e2e-3400-4635-b36d-13b9d05fbccc None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:20.190399 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02030e2e-3400-4635-b36d-13b9d05fbccc None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:20.195036 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1835ecee-7fbd-4809-af8e-3db17fc99eed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:20.195618 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835ecee-7fbd-4809-af8e-3db17fc99eed admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:20.195890 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835ecee-7fbd-4809-af8e-3db17fc99eed admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:20.196162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1835ecee-7fbd-4809-af8e-3db17fc99eed admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:20.196805 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1835ecee-7fbd-4809-af8e-3db17fc99eed admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:12:20.197134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1835ecee-7fbd-4809-af8e-3db17fc99eed admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:20.198736 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1835ecee-7fbd-4809-af8e-3db17fc99eed admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:20.210841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1221/2448] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:12:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:12:20.219284 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666e4b9e-b157-444e-94c4-93e50736d41f None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:20.228468 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666e4b9e-b157-444e-94c4-93e50736d41f None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:20.233209 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef31684c-6ada-4cb3-9247-be37a5b982a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:20.234397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef31684c-6ada-4cb3-9247-be37a5b982a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:20.234670 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef31684c-6ada-4cb3-9247-be37a5b982a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:20.234943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef31684c-6ada-4cb3-9247-be37a5b982a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:20.235579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef31684c-6ada-4cb3-9247-be37a5b982a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:20.237499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef31684c-6ada-4cb3-9247-be37a5b982a1 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:20.366831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1228/2449] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:12:20 2022] POST /identity/v3/projects => generated 367 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:20.373131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1835ecee-7fbd-4809-af8e-3db17fc99eed admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:20.384379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1835ecee-7fbd-4809-af8e-3db17fc99eed admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:20.444865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f40148-f73a-4c3c-85e2-8b5fe0112f71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:20.445517 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f40148-f73a-4c3c-85e2-8b5fe0112f71 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:20.445797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f40148-f73a-4c3c-85e2-8b5fe0112f71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:20.446075 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f40148-f73a-4c3c-85e2-8b5fe0112f71 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:20.446736 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f40148-f73a-4c3c-85e2-8b5fe0112f71 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:20.448711 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f40148-f73a-4c3c-85e2-8b5fe0112f71 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:20.564494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1222/2450] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:12:20 2022] POST /identity/v3/users => generated 284 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:12:20.568771 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef31684c-6ada-4cb3-9247-be37a5b982a1 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:20.579331 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef31684c-6ada-4cb3-9247-be37a5b982a1 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:20.584085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1d21ca-5fab-4267-838f-94ed0f62282e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:20.584774 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1d21ca-5fab-4267-838f-94ed0f62282e admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:20.585032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1d21ca-5fab-4267-838f-94ed0f62282e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:20.585347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1d21ca-5fab-4267-838f-94ed0f62282e admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:20.586020 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1d21ca-5fab-4267-838f-94ed0f62282e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:20.587649 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1d21ca-5fab-4267-838f-94ed0f62282e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:20.597962 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1229/2451] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:20.605814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f40148-f73a-4c3c-85e2-8b5fe0112f71 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:20.616553 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f40148-f73a-4c3c-85e2-8b5fe0112f71 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:20.621450 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b3e9d0-43aa-41c0-93f6-3f8039a7ce6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:20.622044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b3e9d0-43aa-41c0-93f6-3f8039a7ce6a admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:20.622321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b3e9d0-43aa-41c0-93f6-3f8039a7ce6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:20.622591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b3e9d0-43aa-41c0-93f6-3f8039a7ce6a admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:20.623261 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b3e9d0-43aa-41c0-93f6-3f8039a7ce6a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:20.625562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b3e9d0-43aa-41c0-93f6-3f8039a7ce6a admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:20.636074 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1223/2452] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:20.640310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1d21ca-5fab-4267-838f-94ed0f62282e admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:20.650086 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1d21ca-5fab-4267-838f-94ed0f62282e admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:20.654483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab2eb21-dea5-462b-b286-5ba8fcecee07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:20.655135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab2eb21-dea5-462b-b286-5ba8fcecee07 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:20.655414 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab2eb21-dea5-462b-b286-5ba8fcecee07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:20.655686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab2eb21-dea5-462b-b286-5ba8fcecee07 admin admin] PATH_INFO: `/v3/projects/2229967a2f824dd4ad40828726280715/users/f9d1011da63148d2b7971af743a3f2ea/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:20.669995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab2eb21-dea5-462b-b286-5ba8fcecee07 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2229967a2f824dd4ad40828726280715, user_id=f9d1011da63148d2b7971af743a3f2ea, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:20.673729 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab2eb21-dea5-462b-b286-5ba8fcecee07 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:20.746260 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1230/2453] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:20 2022] PUT /identity/v3/projects/2229967a2f824dd4ad40828726280715/users/f9d1011da63148d2b7971af743a3f2ea/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:20.757590 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4712f2-e632-4537-82ed-a74a6cb9320a None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:20.757878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4712f2-e632-4537-82ed-a74a6cb9320a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:20.758182 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4712f2-e632-4537-82ed-a74a6cb9320a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:20.792367 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-8e4712f2-e632-4537-82ed-a74a6cb9320a None None] MFA Rules not processed for user `f9d1011da63148d2b7971af743a3f2ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:20.793619 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8e4712f2-e632-4537-82ed-a74a6cb9320a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:20.822522 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:20.861196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1224/2454] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:20 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:20.865176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab2eb21-dea5-462b-b286-5ba8fcecee07 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:20.928529 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab2eb21-dea5-462b-b286-5ba8fcecee07 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:20.933198 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde6fc71-587b-46cf-bf20-fad26955aea9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:20.933804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde6fc71-587b-46cf-bf20-fad26955aea9 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:20.934087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde6fc71-587b-46cf-bf20-fad26955aea9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:20.934363 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde6fc71-587b-46cf-bf20-fad26955aea9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:20.935003 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde6fc71-587b-46cf-bf20-fad26955aea9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:20.937515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde6fc71-587b-46cf-bf20-fad26955aea9 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:20.942868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:21.032076 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1231/2455] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:12:20 2022] POST /identity/v3/projects => generated 367 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:21.036252 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4712f2-e632-4537-82ed-a74a6cb9320a None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:21.046854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4712f2-e632-4537-82ed-a74a6cb9320a None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:21.103042 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eab2a8b-cd58-4283-8c3c-a320b027fe7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:21.103659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eab2a8b-cd58-4283-8c3c-a320b027fe7b admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:21.103863 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eab2a8b-cd58-4283-8c3c-a320b027fe7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:21.104138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eab2a8b-cd58-4283-8c3c-a320b027fe7b admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:21.104808 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eab2a8b-cd58-4283-8c3c-a320b027fe7b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:21.106746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eab2a8b-cd58-4283-8c3c-a320b027fe7b admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:21.203564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1225/2456] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:12:21 2022] POST /identity/v3/users => generated 290 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:21.207650 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde6fc71-587b-46cf-bf20-fad26955aea9 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:21.218634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde6fc71-587b-46cf-bf20-fad26955aea9 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:21.230432 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb5eda3-c0d8-4373-8983-e4845ea9a2e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:21.230432 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb5eda3-c0d8-4373-8983-e4845ea9a2e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:21.230432 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb5eda3-c0d8-4373-8983-e4845ea9a2e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:21.230432 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb5eda3-c0d8-4373-8983-e4845ea9a2e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:21.230801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb5eda3-c0d8-4373-8983-e4845ea9a2e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:21.230801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb5eda3-c0d8-4373-8983-e4845ea9a2e3 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:21.237276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1232/2457] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:21.241948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eab2a8b-cd58-4283-8c3c-a320b027fe7b admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:21.252943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eab2a8b-cd58-4283-8c3c-a320b027fe7b admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:21.257667 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b47d63-a32b-4616-ab2f-1776a14c264c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:21.258317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b47d63-a32b-4616-ab2f-1776a14c264c admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:21.258627 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b47d63-a32b-4616-ab2f-1776a14c264c admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:21.258904 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b47d63-a32b-4616-ab2f-1776a14c264c admin admin] PATH_INFO: `/v3/projects/9c0b331a0c17484f89a5f40f21f1ee4b/users/9efe45e0ac044f6a9fbf8b93641f16ee/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:21.273629 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b47d63-a32b-4616-ab2f-1776a14c264c admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c0b331a0c17484f89a5f40f21f1ee4b, user_id=9efe45e0ac044f6a9fbf8b93641f16ee, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:21.276264 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b47d63-a32b-4616-ab2f-1776a14c264c admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:21.322238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1226/2458] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:21 2022] PUT /identity/v3/projects/9c0b331a0c17484f89a5f40f21f1ee4b/users/9efe45e0ac044f6a9fbf8b93641f16ee/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:12:21.330552 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb5eda3-c0d8-4373-8983-e4845ea9a2e3 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:21.340625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb5eda3-c0d8-4373-8983-e4845ea9a2e3 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:21.397162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f35e8a-8387-4c84-be6d-fb2604ee8426 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:21.397686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f35e8a-8387-4c84-be6d-fb2604ee8426 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:21.397968 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f35e8a-8387-4c84-be6d-fb2604ee8426 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:21.398244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f35e8a-8387-4c84-be6d-fb2604ee8426 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:21.398887 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f35e8a-8387-4c84-be6d-fb2604ee8426 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:21.400496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f35e8a-8387-4c84-be6d-fb2604ee8426 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:21.410644 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1233/2459] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:21 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:21.415025 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b47d63-a32b-4616-ab2f-1776a14c264c admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:21.425959 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b47d63-a32b-4616-ab2f-1776a14c264c admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:21.430517 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b4d73f-57d9-4393-bb09-3b1428531f4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:21.431159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b4d73f-57d9-4393-bb09-3b1428531f4c admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:21.431434 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b4d73f-57d9-4393-bb09-3b1428531f4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:21.431708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b4d73f-57d9-4393-bb09-3b1428531f4c admin admin] PATH_INFO: `/v3/projects/9c0b331a0c17484f89a5f40f21f1ee4b/users/9efe45e0ac044f6a9fbf8b93641f16ee/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:21.433645 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b4d73f-57d9-4393-bb09-3b1428531f4c admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c0b331a0c17484f89a5f40f21f1ee4b, user_id=9efe45e0ac044f6a9fbf8b93641f16ee, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:21.436250 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b4d73f-57d9-4393-bb09-3b1428531f4c admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:21.447279 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1227/2460] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:21 2022] PUT /identity/v3/projects/9c0b331a0c17484f89a5f40f21f1ee4b/users/9efe45e0ac044f6a9fbf8b93641f16ee/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:21.458794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d29c30-635c-4d42-9e86-7acc79bd80b4 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:21.459081 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d29c30-635c-4d42-9e86-7acc79bd80b4 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:21.459357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d29c30-635c-4d42-9e86-7acc79bd80b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:21.489487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-79d29c30-635c-4d42-9e86-7acc79bd80b4 None None] MFA Rules not processed for user `9efe45e0ac044f6a9fbf8b93641f16ee`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:21.490704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-79d29c30-635c-4d42-9e86-7acc79bd80b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:21.560357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1234/2461] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:21 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:21.574634 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b4d73f-57d9-4393-bb09-3b1428531f4c admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:21.576422 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f0b4d73f-57d9-4393-bb09-3b1428531f4c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:21.636811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b4d73f-57d9-4393-bb09-3b1428531f4c admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:21.640562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd202dea-2c16-49b5-89b0-b25d239ae905 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:21.641198 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd202dea-2c16-49b5-89b0-b25d239ae905 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:21.641478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd202dea-2c16-49b5-89b0-b25d239ae905 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:21.641807 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd202dea-2c16-49b5-89b0-b25d239ae905 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:21.652446 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd202dea-2c16-49b5-89b0-b25d239ae905 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:21.653962 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd202dea-2c16-49b5-89b0-b25d239ae905 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:21.663996 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1228/2462] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:21 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:21.777998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d29c30-635c-4d42-9e86-7acc79bd80b4 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:21.779413 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-79d29c30-635c-4d42-9e86-7acc79bd80b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:21.791222 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d29c30-635c-4d42-9e86-7acc79bd80b4 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:21.794951 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bb1300-a715-4353-9fe6-b156211398a0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:21.795562 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bb1300-a715-4353-9fe6-b156211398a0 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:21.795849 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bb1300-a715-4353-9fe6-b156211398a0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:21.796187 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bb1300-a715-4353-9fe6-b156211398a0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:21.850191 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:21.852460 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bb1300-a715-4353-9fe6-b156211398a0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:21.854006 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bb1300-a715-4353-9fe6-b156211398a0 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:21.862755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1235/2463] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:21 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:26.566308 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd202dea-2c16-49b5-89b0-b25d239ae905 service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:26.593495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:26.628925 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd202dea-2c16-49b5-89b0-b25d239ae905 service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:26.633430 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea07b0b0-7efc-4658-b66d-4bf289f3b4ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:26.634067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea07b0b0-7efc-4658-b66d-4bf289f3b4ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:26.634321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea07b0b0-7efc-4658-b66d-4bf289f3b4ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:26.634587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea07b0b0-7efc-4658-b66d-4bf289f3b4ef admin admin] PATH_INFO: `/v3/users/f9d1011da63148d2b7971af743a3f2ea` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:26.635750 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea07b0b0-7efc-4658-b66d-4bf289f3b4ef admin admin] RBAC: Authorizing `identity:delete_user(user_id=f9d1011da63148d2b7971af743a3f2ea)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:26.637703 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea07b0b0-7efc-4658-b66d-4bf289f3b4ef admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:26.811533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ea07b0b0-7efc-4658-b66d-4bf289f3b4ef admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f9d1011da63148d2b7971af743a3f2ea'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:26.812189 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ea07b0b0-7efc-4658-b66d-4bf289f3b4ef admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f9d1011da63148d2b7971af743a3f2ea'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:26.865660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ea07b0b0-7efc-4658-b66d-4bf289f3b4ef admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f9d1011da63148d2b7971af743a3f2ea'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:26.878292 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ea07b0b0-7efc-4658-b66d-4bf289f3b4ef admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f9d1011da63148d2b7971af743a3f2ea'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:26.878924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ea07b0b0-7efc-4658-b66d-4bf289f3b4ef admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f9d1011da63148d2b7971af743a3f2ea'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:26.890997 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1229/2464] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:26 2022] DELETE /identity/v3/users/f9d1011da63148d2b7971af743a3f2ea => generated 0 bytes in 325 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:26.895511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23772177-7ecb-455f-a867-c9e846d99110 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:26.896111 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23772177-7ecb-455f-a867-c9e846d99110 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:26.896111 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23772177-7ecb-455f-a867-c9e846d99110 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:26.915838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-23772177-7ecb-455f-a867-c9e846d99110 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:26.917123 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-23772177-7ecb-455f-a867-c9e846d99110 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:26.975942 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1236/2465] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:12:26.985685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea07b0b0-7efc-4658-b66d-4bf289f3b4ef admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:26.987137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ea07b0b0-7efc-4658-b66d-4bf289f3b4ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:27.043141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea07b0b0-7efc-4658-b66d-4bf289f3b4ef admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:27.046768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d00c9b8-78a5-4d5e-81d2-d56494a5ded8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:27.047537 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d00c9b8-78a5-4d5e-81d2-d56494a5ded8 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:27.047809 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d00c9b8-78a5-4d5e-81d2-d56494a5ded8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:27.048077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d00c9b8-78a5-4d5e-81d2-d56494a5ded8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:27.056765 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d00c9b8-78a5-4d5e-81d2-d56494a5ded8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:27.058185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d00c9b8-78a5-4d5e-81d2-d56494a5ded8 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:27.069697 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1230/2466] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:27.852957 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23772177-7ecb-455f-a867-c9e846d99110 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:27.852957 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-23772177-7ecb-455f-a867-c9e846d99110 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:27.861254 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23772177-7ecb-455f-a867-c9e846d99110 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:27.866659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:27.867389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:27.867732 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:27.868066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf admin admin] PATH_INFO: `/v3/projects/2229967a2f824dd4ad40828726280715` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:27.869341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf admin admin] RBAC: Authorizing `identity:delete_project(project_id=2229967a2f824dd4ad40828726280715)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:27.871351 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:28.011163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2229967a2f824dd4ad40828726280715', 'request_id': 'req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:28.017739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2229967a2f824dd4ad40828726280715', 'request_id': 'req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:28.024270 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2229967a2f824dd4ad40828726280715', 'request_id': 'req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:28.151420 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf admin admin] The token cache is being invalidate because project 2229967a2f824dd4ad40828726280715 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:12:28.151714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:28.152335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:28.153850 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1237/2467] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:12:27 2022] DELETE /identity/v3/projects/2229967a2f824dd4ad40828726280715 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:28.157909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d00c9b8-78a5-4d5e-81d2-d56494a5ded8 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:28.167269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d00c9b8-78a5-4d5e-81d2-d56494a5ded8 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:28.214605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:28.228049 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33de8703-c568-44be-a96b-810a0acc6969 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:28.228681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33de8703-c568-44be-a96b-810a0acc6969 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:28.228961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33de8703-c568-44be-a96b-810a0acc6969 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:28.229264 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33de8703-c568-44be-a96b-810a0acc6969 admin admin] PATH_INFO: `/v3/users/9efe45e0ac044f6a9fbf8b93641f16ee` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:28.230404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33de8703-c568-44be-a96b-810a0acc6969 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9efe45e0ac044f6a9fbf8b93641f16ee)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:28.232313 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33de8703-c568-44be-a96b-810a0acc6969 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:28.407396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-33de8703-c568-44be-a96b-810a0acc6969 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9efe45e0ac044f6a9fbf8b93641f16ee'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:28.408096 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-33de8703-c568-44be-a96b-810a0acc6969 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9efe45e0ac044f6a9fbf8b93641f16ee'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:28.477734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-33de8703-c568-44be-a96b-810a0acc6969 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9efe45e0ac044f6a9fbf8b93641f16ee'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:28.492513 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-33de8703-c568-44be-a96b-810a0acc6969 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9efe45e0ac044f6a9fbf8b93641f16ee'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:28.493376 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-33de8703-c568-44be-a96b-810a0acc6969 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9efe45e0ac044f6a9fbf8b93641f16ee'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:28.507148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1231/2468] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:28 2022] DELETE /identity/v3/users/9efe45e0ac044f6a9fbf8b93641f16ee => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:29.076104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:29.077848 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:29.102951 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e66cbe9-9de4-4ed2-bf6d-b8e839e0f6bf admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:29.180401 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6ae70c-73c7-4a0c-a3d5-42365a7cd7b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:29.181402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6ae70c-73c7-4a0c-a3d5-42365a7cd7b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:29.181749 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6ae70c-73c7-4a0c-a3d5-42365a7cd7b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:29.182105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6ae70c-73c7-4a0c-a3d5-42365a7cd7b1 admin admin] PATH_INFO: `/v3/projects/9c0b331a0c17484f89a5f40f21f1ee4b` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:29.183354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6ae70c-73c7-4a0c-a3d5-42365a7cd7b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9c0b331a0c17484f89a5f40f21f1ee4b)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:29.185539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6ae70c-73c7-4a0c-a3d5-42365a7cd7b1 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:29.214815 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:29.319140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cb6ae70c-73c7-4a0c-a3d5-42365a7cd7b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9c0b331a0c17484f89a5f40f21f1ee4b', 'request_id': 'req-cb6ae70c-73c7-4a0c-a3d5-42365a7cd7b1'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:29.325297 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cb6ae70c-73c7-4a0c-a3d5-42365a7cd7b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9c0b331a0c17484f89a5f40f21f1ee4b', 'request_id': 'req-cb6ae70c-73c7-4a0c-a3d5-42365a7cd7b1'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:29.332999 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cb6ae70c-73c7-4a0c-a3d5-42365a7cd7b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9c0b331a0c17484f89a5f40f21f1ee4b', 'request_id': 'req-cb6ae70c-73c7-4a0c-a3d5-42365a7cd7b1'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:29.435993 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cb6ae70c-73c7-4a0c-a3d5-42365a7cd7b1 admin admin] The token cache is being invalidate because project 9c0b331a0c17484f89a5f40f21f1ee4b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:12:29.436546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cb6ae70c-73c7-4a0c-a3d5-42365a7cd7b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:29.437207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cb6ae70c-73c7-4a0c-a3d5-42365a7cd7b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:29.438832 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1238/2469] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:12:29 2022] DELETE /identity/v3/projects/9c0b331a0c17484f89a5f40f21f1ee4b => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:29.443814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20abbf1-9a09-4783-9704-96a2346ff121 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:29.444356 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20abbf1-9a09-4783-9704-96a2346ff121 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:29.444536 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20abbf1-9a09-4783-9704-96a2346ff121 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:29.477943 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-b20abbf1-9a09-4783-9704-96a2346ff121 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:29.479172 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b20abbf1-9a09-4783-9704-96a2346ff121 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:29.552297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1232/2470] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:29.560077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7e96ec-60af-478b-bac5-ba0c2184b5a0 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:29.560426 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7e96ec-60af-478b-bac5-ba0c2184b5a0 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:29.560811 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7e96ec-60af-478b-bac5-ba0c2184b5a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:29.582819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-ed7e96ec-60af-478b-bac5-ba0c2184b5a0 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:29.584160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ed7e96ec-60af-478b-bac5-ba0c2184b5a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:29.593497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1239/2471] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:29.600599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20abbf1-9a09-4783-9704-96a2346ff121 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:29.615038 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20abbf1-9a09-4783-9704-96a2346ff121 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:29.620483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5e8653-d73e-47e3-aa8f-659abc77ef5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:29.621112 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5e8653-d73e-47e3-aa8f-659abc77ef5e admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:29.621441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5e8653-d73e-47e3-aa8f-659abc77ef5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:29.621722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5e8653-d73e-47e3-aa8f-659abc77ef5e admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:29.622369 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5e8653-d73e-47e3-aa8f-659abc77ef5e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:12:29.622670 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5e8653-d73e-47e3-aa8f-659abc77ef5e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:29.624300 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f5e8653-d73e-47e3-aa8f-659abc77ef5e admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:29.637376 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1233/2472] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:12:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:12:29.643279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7e96ec-60af-478b-bac5-ba0c2184b5a0 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:29.653288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7e96ec-60af-478b-bac5-ba0c2184b5a0 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:29.659099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71d8bfa-6275-4433-9e2c-d53acc4b5251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:29.659703 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71d8bfa-6275-4433-9e2c-d53acc4b5251 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:29.659984 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71d8bfa-6275-4433-9e2c-d53acc4b5251 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:29.660257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71d8bfa-6275-4433-9e2c-d53acc4b5251 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:29.660890 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71d8bfa-6275-4433-9e2c-d53acc4b5251 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:29.662856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71d8bfa-6275-4433-9e2c-d53acc4b5251 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:29.756238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1240/2473] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:12:29 2022] POST /identity/v3/projects => generated 399 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:29.760358 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5e8653-d73e-47e3-aa8f-659abc77ef5e admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:29.772289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5e8653-d73e-47e3-aa8f-659abc77ef5e admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:29.842388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:29.851204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c3b432-e7ea-4a7d-8679-4c288a67e8e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:29.851975 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c3b432-e7ea-4a7d-8679-4c288a67e8e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:29.860286 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c3b432-e7ea-4a7d-8679-4c288a67e8e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:29.860286 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c3b432-e7ea-4a7d-8679-4c288a67e8e0 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:29.860286 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c3b432-e7ea-4a7d-8679-4c288a67e8e0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:29.860286 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c3b432-e7ea-4a7d-8679-4c288a67e8e0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:29.946392 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1234/2474] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:12:29 2022] POST /identity/v3/users => generated 300 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:12:29.950777 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71d8bfa-6275-4433-9e2c-d53acc4b5251 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:29.962201 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71d8bfa-6275-4433-9e2c-d53acc4b5251 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:29.967025 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae6356c-3bbf-4eeb-b3d5-93260c7230fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:29.967646 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae6356c-3bbf-4eeb-b3d5-93260c7230fc admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:29.967946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae6356c-3bbf-4eeb-b3d5-93260c7230fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:29.968221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae6356c-3bbf-4eeb-b3d5-93260c7230fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:29.969422 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae6356c-3bbf-4eeb-b3d5-93260c7230fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:29.971285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae6356c-3bbf-4eeb-b3d5-93260c7230fc admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:29.982074 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1241/2475] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:29.986797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c3b432-e7ea-4a7d-8679-4c288a67e8e0 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:29.998162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c3b432-e7ea-4a7d-8679-4c288a67e8e0 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:30.003285 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0343d6ea-45ef-4b32-99a9-808597773ef5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:30.004042 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0343d6ea-45ef-4b32-99a9-808597773ef5 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:30.004323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0343d6ea-45ef-4b32-99a9-808597773ef5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:30.004594 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0343d6ea-45ef-4b32-99a9-808597773ef5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:30.005291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0343d6ea-45ef-4b32-99a9-808597773ef5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:30.006989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0343d6ea-45ef-4b32-99a9-808597773ef5 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:30.018610 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1235/2476] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:12:30.022872 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae6356c-3bbf-4eeb-b3d5-93260c7230fc admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:30.038036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae6356c-3bbf-4eeb-b3d5-93260c7230fc admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:30.042927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9635179-19ee-4be2-8a81-f5243be3168c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:30.043726 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9635179-19ee-4be2-8a81-f5243be3168c admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:30.044009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9635179-19ee-4be2-8a81-f5243be3168c admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:30.044284 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9635179-19ee-4be2-8a81-f5243be3168c admin admin] PATH_INFO: `/v3/projects/793bd52dc1be46888c6a54c22a46be0a/users/0bea3e9f122a49f7a06626f936ca0fb7/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:30.062992 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9635179-19ee-4be2-8a81-f5243be3168c admin admin] RBAC: Authorizing `identity:create_grant(project_id=793bd52dc1be46888c6a54c22a46be0a, user_id=0bea3e9f122a49f7a06626f936ca0fb7, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:30.066039 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9635179-19ee-4be2-8a81-f5243be3168c admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:30.142444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1242/2477] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:30 2022] PUT /identity/v3/projects/793bd52dc1be46888c6a54c22a46be0a/users/0bea3e9f122a49f7a06626f936ca0fb7/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:30.153095 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669e78cc-663c-4ae4-8344-6c9ce685128b None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:30.153402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669e78cc-663c-4ae4-8344-6c9ce685128b None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:30.153676 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669e78cc-663c-4ae4-8344-6c9ce685128b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:30.185926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-669e78cc-663c-4ae4-8344-6c9ce685128b None None] MFA Rules not processed for user `0bea3e9f122a49f7a06626f936ca0fb7`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:30.187136 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-669e78cc-663c-4ae4-8344-6c9ce685128b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:30.260441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1236/2478] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:30 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:30.265169 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9635179-19ee-4be2-8a81-f5243be3168c admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:30.334210 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9635179-19ee-4be2-8a81-f5243be3168c admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:30.338991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78356de-49e6-4b3a-8089-8fb7cbf35b8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:30.339600 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78356de-49e6-4b3a-8089-8fb7cbf35b8f admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:30.339909 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78356de-49e6-4b3a-8089-8fb7cbf35b8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:30.340163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78356de-49e6-4b3a-8089-8fb7cbf35b8f admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:30.340809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78356de-49e6-4b3a-8089-8fb7cbf35b8f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:30.342774 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78356de-49e6-4b3a-8089-8fb7cbf35b8f admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:30.405127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1243/2479] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:12:30 2022] POST /identity/v3/projects => generated 399 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:12:30.412993 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669e78cc-663c-4ae4-8344-6c9ce685128b None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:30.424794 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669e78cc-663c-4ae4-8344-6c9ce685128b None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:30.483615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e141694-6539-4b24-9584-599c0e5892d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:30.484237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e141694-6539-4b24-9584-599c0e5892d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:30.484539 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e141694-6539-4b24-9584-599c0e5892d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:30.484891 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e141694-6539-4b24-9584-599c0e5892d2 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:30.485560 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e141694-6539-4b24-9584-599c0e5892d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:30.487469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e141694-6539-4b24-9584-599c0e5892d2 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:30.551699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1237/2480] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:12:30 2022] POST /identity/v3/users => generated 306 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:30.555716 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78356de-49e6-4b3a-8089-8fb7cbf35b8f admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:30.566583 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78356de-49e6-4b3a-8089-8fb7cbf35b8f admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:30.571190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5055d698-732c-44be-8a37-8db123492261 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:30.571779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5055d698-732c-44be-8a37-8db123492261 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:30.572066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5055d698-732c-44be-8a37-8db123492261 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:30.572340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5055d698-732c-44be-8a37-8db123492261 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:30.572982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5055d698-732c-44be-8a37-8db123492261 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:30.574613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5055d698-732c-44be-8a37-8db123492261 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:30.586874 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1244/2481] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:30.591083 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e141694-6539-4b24-9584-599c0e5892d2 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:30.601023 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e141694-6539-4b24-9584-599c0e5892d2 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:30.605614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de049f1-5b20-47e1-90e4-af5014d55892 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:30.606254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de049f1-5b20-47e1-90e4-af5014d55892 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:30.606530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de049f1-5b20-47e1-90e4-af5014d55892 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:30.606800 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de049f1-5b20-47e1-90e4-af5014d55892 admin admin] PATH_INFO: `/v3/projects/e1b11403119042b293cce37def798c23/users/59884b9c95e44318b25c1426ddcccf76/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:30.621725 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de049f1-5b20-47e1-90e4-af5014d55892 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1b11403119042b293cce37def798c23, user_id=59884b9c95e44318b25c1426ddcccf76, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:30.624351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de049f1-5b20-47e1-90e4-af5014d55892 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:30.706418 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1238/2482] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:30 2022] PUT /identity/v3/projects/e1b11403119042b293cce37def798c23/users/59884b9c95e44318b25c1426ddcccf76/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:30.710370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5055d698-732c-44be-8a37-8db123492261 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:30.721285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5055d698-732c-44be-8a37-8db123492261 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:30.740268 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:30.780504 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7858f751-212f-444d-9750-4c9fd788a8d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:30.781654 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7858f751-212f-444d-9750-4c9fd788a8d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:30.781956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7858f751-212f-444d-9750-4c9fd788a8d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:30.782219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7858f751-212f-444d-9750-4c9fd788a8d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:30.782855 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7858f751-212f-444d-9750-4c9fd788a8d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:30.784432 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7858f751-212f-444d-9750-4c9fd788a8d3 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:30.794852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1245/2483] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:30.799289 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de049f1-5b20-47e1-90e4-af5014d55892 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:30.809520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de049f1-5b20-47e1-90e4-af5014d55892 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:30.815051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c3dd38-c3cf-48b5-9aca-32e2c1d7a3b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:30.815325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c3dd38-c3cf-48b5-9aca-32e2c1d7a3b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:30.815601 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c3dd38-c3cf-48b5-9aca-32e2c1d7a3b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:30.815869 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c3dd38-c3cf-48b5-9aca-32e2c1d7a3b9 admin admin] PATH_INFO: `/v3/projects/e1b11403119042b293cce37def798c23/users/59884b9c95e44318b25c1426ddcccf76/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:30.817830 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c3dd38-c3cf-48b5-9aca-32e2c1d7a3b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1b11403119042b293cce37def798c23, user_id=59884b9c95e44318b25c1426ddcccf76, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:30.820463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c3dd38-c3cf-48b5-9aca-32e2c1d7a3b9 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:30.831238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1239/2484] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:30 2022] PUT /identity/v3/projects/e1b11403119042b293cce37def798c23/users/59884b9c95e44318b25c1426ddcccf76/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:30.842429 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d55795-c83e-40dc-b7f8-15476a7b1079 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:30.842722 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d55795-c83e-40dc-b7f8-15476a7b1079 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:30.843001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d55795-c83e-40dc-b7f8-15476a7b1079 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:30.873224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-08d55795-c83e-40dc-b7f8-15476a7b1079 None None] MFA Rules not processed for user `59884b9c95e44318b25c1426ddcccf76`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:30.874430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-08d55795-c83e-40dc-b7f8-15476a7b1079 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:30.949145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1246/2485] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:30 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:30.959818 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c3dd38-c3cf-48b5-9aca-32e2c1d7a3b9 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:30.961609 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-93c3dd38-c3cf-48b5-9aca-32e2c1d7a3b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:30.976902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:31.022257 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c3dd38-c3cf-48b5-9aca-32e2c1d7a3b9 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:31.026027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9f4de2-21a3-4911-8021-8bb718a717cc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:31.028001 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9f4de2-21a3-4911-8021-8bb718a717cc service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:31.028001 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9f4de2-21a3-4911-8021-8bb718a717cc service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:31.028001 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9f4de2-21a3-4911-8021-8bb718a717cc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:31.037417 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9f4de2-21a3-4911-8021-8bb718a717cc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:31.038865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9f4de2-21a3-4911-8021-8bb718a717cc service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:31.050445 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1240/2486] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:30 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:31.268128 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d55795-c83e-40dc-b7f8-15476a7b1079 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:31.278267 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d55795-c83e-40dc-b7f8-15476a7b1079 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:31.282852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba62d9c-3933-4b9b-aa10-a4f077380d3e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:31.283552 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba62d9c-3933-4b9b-aa10-a4f077380d3e service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:31.283834 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba62d9c-3933-4b9b-aa10-a4f077380d3e service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:31.284253 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba62d9c-3933-4b9b-aa10-a4f077380d3e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:31.342505 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba62d9c-3933-4b9b-aa10-a4f077380d3e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:31.344633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba62d9c-3933-4b9b-aa10-a4f077380d3e service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:31.361368 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1247/2487] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:31 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:35.948032 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9f4de2-21a3-4911-8021-8bb718a717cc service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:35.949458 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-4f9f4de2-21a3-4911-8021-8bb718a717cc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:36.009215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9f4de2-21a3-4911-8021-8bb718a717cc service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:36.013132 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dfc822-c7d0-46a2-9c47-f6a0007a4039 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:36.013735 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dfc822-c7d0-46a2-9c47-f6a0007a4039 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:36.014008 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dfc822-c7d0-46a2-9c47-f6a0007a4039 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:36.014272 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dfc822-c7d0-46a2-9c47-f6a0007a4039 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:36.016043 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d4dfc822-c7d0-46a2-9c47-f6a0007a4039 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:36.082594 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dfc822-c7d0-46a2-9c47-f6a0007a4039 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:36.084052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dfc822-c7d0-46a2-9c47-f6a0007a4039 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:36.094205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1241/2488] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:36.322464 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba62d9c-3933-4b9b-aa10-a4f077380d3e service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:36.333571 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba62d9c-3933-4b9b-aa10-a4f077380d3e service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:36.338069 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e814b46-578f-47f9-b94c-ee7aaaadf8a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:36.338691 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e814b46-578f-47f9-b94c-ee7aaaadf8a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:36.338972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e814b46-578f-47f9-b94c-ee7aaaadf8a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:36.339438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e814b46-578f-47f9-b94c-ee7aaaadf8a2 admin admin] PATH_INFO: `/v3/users/0bea3e9f122a49f7a06626f936ca0fb7` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:36.340582 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e814b46-578f-47f9-b94c-ee7aaaadf8a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0bea3e9f122a49f7a06626f936ca0fb7)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:36.342533 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e814b46-578f-47f9-b94c-ee7aaaadf8a2 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:36.471255 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:36.480950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4e814b46-578f-47f9-b94c-ee7aaaadf8a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0bea3e9f122a49f7a06626f936ca0fb7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:36.556893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4e814b46-578f-47f9-b94c-ee7aaaadf8a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0bea3e9f122a49f7a06626f936ca0fb7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:36.570664 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4e814b46-578f-47f9-b94c-ee7aaaadf8a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0bea3e9f122a49f7a06626f936ca0fb7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:36.571349 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4e814b46-578f-47f9-b94c-ee7aaaadf8a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0bea3e9f122a49f7a06626f936ca0fb7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:36.578636 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4e814b46-578f-47f9-b94c-ee7aaaadf8a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0bea3e9f122a49f7a06626f936ca0fb7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:36.584549 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1248/2489] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:36 2022] DELETE /identity/v3/users/0bea3e9f122a49f7a06626f936ca0fb7 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:36.589024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40189928-17ba-4645-995d-030d899a5004 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:36.589338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40189928-17ba-4645-995d-030d899a5004 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:36.589613 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40189928-17ba-4645-995d-030d899a5004 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:36.609780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-40189928-17ba-4645-995d-030d899a5004 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:36.611567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-40189928-17ba-4645-995d-030d899a5004 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:36.669908 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1242/2490] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:12:36.680539 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e814b46-578f-47f9-b94c-ee7aaaadf8a2 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:36.682003 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4e814b46-578f-47f9-b94c-ee7aaaadf8a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:36.741059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e814b46-578f-47f9-b94c-ee7aaaadf8a2 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:36.745323 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feee10a9-8018-4e68-be8d-d69ee446e69c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:36.745951 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feee10a9-8018-4e68-be8d-d69ee446e69c service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:36.746234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feee10a9-8018-4e68-be8d-d69ee446e69c service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:36.746511 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feee10a9-8018-4e68-be8d-d69ee446e69c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:36.755921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feee10a9-8018-4e68-be8d-d69ee446e69c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:36.757397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feee10a9-8018-4e68-be8d-d69ee446e69c service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:36.766912 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1249/2491] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:37.161502 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40189928-17ba-4645-995d-030d899a5004 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:37.163620 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-40189928-17ba-4645-995d-030d899a5004 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:37.178141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40189928-17ba-4645-995d-030d899a5004 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:37.185546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39958386-ec5b-4b06-97f6-8b04803dda6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:37.186932 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39958386-ec5b-4b06-97f6-8b04803dda6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:37.190327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39958386-ec5b-4b06-97f6-8b04803dda6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:37.190327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39958386-ec5b-4b06-97f6-8b04803dda6a admin admin] PATH_INFO: `/v3/projects/793bd52dc1be46888c6a54c22a46be0a` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:37.190327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39958386-ec5b-4b06-97f6-8b04803dda6a admin admin] RBAC: Authorizing `identity:delete_project(project_id=793bd52dc1be46888c6a54c22a46be0a)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:37.192319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39958386-ec5b-4b06-97f6-8b04803dda6a admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:37.199447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:37.312351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-39958386-ec5b-4b06-97f6-8b04803dda6a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '793bd52dc1be46888c6a54c22a46be0a', 'request_id': 'req-39958386-ec5b-4b06-97f6-8b04803dda6a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:37.318772 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-39958386-ec5b-4b06-97f6-8b04803dda6a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '793bd52dc1be46888c6a54c22a46be0a', 'request_id': 'req-39958386-ec5b-4b06-97f6-8b04803dda6a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:37.363064 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-39958386-ec5b-4b06-97f6-8b04803dda6a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '793bd52dc1be46888c6a54c22a46be0a', 'request_id': 'req-39958386-ec5b-4b06-97f6-8b04803dda6a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:37.375279 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-39958386-ec5b-4b06-97f6-8b04803dda6a admin admin] The token cache is being invalidate because project 793bd52dc1be46888c6a54c22a46be0a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:12:37.375575 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-39958386-ec5b-4b06-97f6-8b04803dda6a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:37.376853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-39958386-ec5b-4b06-97f6-8b04803dda6a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:37.379130 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1243/2492] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:12:37 2022] DELETE /identity/v3/projects/793bd52dc1be46888c6a54c22a46be0a => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:37.383281 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feee10a9-8018-4e68-be8d-d69ee446e69c service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:37.393725 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feee10a9-8018-4e68-be8d-d69ee446e69c service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:37.452218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c429ad-f4c9-46e6-a853-6d1a118e7cf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:37.452853 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c429ad-f4c9-46e6-a853-6d1a118e7cf8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:37.453163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c429ad-f4c9-46e6-a853-6d1a118e7cf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:37.453438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c429ad-f4c9-46e6-a853-6d1a118e7cf8 admin admin] PATH_INFO: `/v3/users/59884b9c95e44318b25c1426ddcccf76` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:37.455354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c429ad-f4c9-46e6-a853-6d1a118e7cf8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=59884b9c95e44318b25c1426ddcccf76)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:37.457315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c429ad-f4c9-46e6-a853-6d1a118e7cf8 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:37.623290 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-12c429ad-f4c9-46e6-a853-6d1a118e7cf8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '59884b9c95e44318b25c1426ddcccf76'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:37.717715 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-12c429ad-f4c9-46e6-a853-6d1a118e7cf8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '59884b9c95e44318b25c1426ddcccf76'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:37.721657 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:37.734927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-12c429ad-f4c9-46e6-a853-6d1a118e7cf8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '59884b9c95e44318b25c1426ddcccf76'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:37.736473 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-12c429ad-f4c9-46e6-a853-6d1a118e7cf8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '59884b9c95e44318b25c1426ddcccf76'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:37.743874 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-12c429ad-f4c9-46e6-a853-6d1a118e7cf8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '59884b9c95e44318b25c1426ddcccf76'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:37.749909 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1250/2493] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:37 2022] DELETE /identity/v3/users/59884b9c95e44318b25c1426ddcccf76 => generated 0 bytes in 367 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:38.285328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39958386-ec5b-4b06-97f6-8b04803dda6a admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:38.286907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-39958386-ec5b-4b06-97f6-8b04803dda6a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:38.303327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39958386-ec5b-4b06-97f6-8b04803dda6a admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:38.374314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c457d503-7e13-4a94-bec7-83337c6d5fc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:38.375069 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c457d503-7e13-4a94-bec7-83337c6d5fc2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:38.375424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c457d503-7e13-4a94-bec7-83337c6d5fc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:38.375843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c457d503-7e13-4a94-bec7-83337c6d5fc2 admin admin] PATH_INFO: `/v3/projects/e1b11403119042b293cce37def798c23` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:38.377825 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c457d503-7e13-4a94-bec7-83337c6d5fc2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e1b11403119042b293cce37def798c23)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:38.379801 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c457d503-7e13-4a94-bec7-83337c6d5fc2 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:38.495595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c457d503-7e13-4a94-bec7-83337c6d5fc2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1b11403119042b293cce37def798c23', 'request_id': 'req-c457d503-7e13-4a94-bec7-83337c6d5fc2'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:38.501442 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c457d503-7e13-4a94-bec7-83337c6d5fc2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1b11403119042b293cce37def798c23', 'request_id': 'req-c457d503-7e13-4a94-bec7-83337c6d5fc2'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:38.557451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c457d503-7e13-4a94-bec7-83337c6d5fc2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1b11403119042b293cce37def798c23', 'request_id': 'req-c457d503-7e13-4a94-bec7-83337c6d5fc2'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:38.569520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c457d503-7e13-4a94-bec7-83337c6d5fc2 admin admin] The token cache is being invalidate because project e1b11403119042b293cce37def798c23 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:12:38.569806 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c457d503-7e13-4a94-bec7-83337c6d5fc2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:38.570432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c457d503-7e13-4a94-bec7-83337c6d5fc2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:38.571884 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1244/2494] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:12:38 2022] DELETE /identity/v3/projects/e1b11403119042b293cce37def798c23 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:38.576942 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88981ad0-9f8a-4814-95c5-b640a1c830d8 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:38.577462 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88981ad0-9f8a-4814-95c5-b640a1c830d8 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:38.577557 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88981ad0-9f8a-4814-95c5-b640a1c830d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:38.601583 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-88981ad0-9f8a-4814-95c5-b640a1c830d8 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:38.602839 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-88981ad0-9f8a-4814-95c5-b640a1c830d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:38.667013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1251/2495] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:38.675226 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3528b1e-e3bc-4702-b5c3-71f9818bcff3 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:38.675510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3528b1e-e3bc-4702-b5c3-71f9818bcff3 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:38.675783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3528b1e-e3bc-4702-b5c3-71f9818bcff3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:38.697154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-a3528b1e-e3bc-4702-b5c3-71f9818bcff3 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:38.698302 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a3528b1e-e3bc-4702-b5c3-71f9818bcff3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:38.706961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1245/2496] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:12:38.711171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88981ad0-9f8a-4814-95c5-b640a1c830d8 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:38.721541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88981ad0-9f8a-4814-95c5-b640a1c830d8 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:38.726431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4e2546-6feb-4072-9510-7607fecc9725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:38.727058 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4e2546-6feb-4072-9510-7607fecc9725 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:38.727333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4e2546-6feb-4072-9510-7607fecc9725 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:38.727601 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4e2546-6feb-4072-9510-7607fecc9725 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:38.728256 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4e2546-6feb-4072-9510-7607fecc9725 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:12:38.728553 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4e2546-6feb-4072-9510-7607fecc9725 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:38.730300 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4e2546-6feb-4072-9510-7607fecc9725 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:38.742805 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1252/2497] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:12:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:12:38.758925 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3528b1e-e3bc-4702-b5c3-71f9818bcff3 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:38.773407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3528b1e-e3bc-4702-b5c3-71f9818bcff3 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:38.785129 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369b0ae8-7a01-4c0c-be1a-6b241cb3b679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:38.785129 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369b0ae8-7a01-4c0c-be1a-6b241cb3b679 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:38.785129 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369b0ae8-7a01-4c0c-be1a-6b241cb3b679 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:38.785129 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369b0ae8-7a01-4c0c-be1a-6b241cb3b679 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:38.785712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369b0ae8-7a01-4c0c-be1a-6b241cb3b679 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:38.785712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369b0ae8-7a01-4c0c-be1a-6b241cb3b679 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:38.798940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:38.907826 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1246/2498] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:12:38 2022] POST /identity/v3/projects => generated 381 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:12:38.912212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4e2546-6feb-4072-9510-7607fecc9725 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:38.926015 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4e2546-6feb-4072-9510-7607fecc9725 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:38.994235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72088218-6645-49a7-8ab2-e58574df5e41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:38.994955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72088218-6645-49a7-8ab2-e58574df5e41 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:38.995305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72088218-6645-49a7-8ab2-e58574df5e41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:38.995646 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72088218-6645-49a7-8ab2-e58574df5e41 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:38.996537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72088218-6645-49a7-8ab2-e58574df5e41 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:38.998741 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72088218-6645-49a7-8ab2-e58574df5e41 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:39.105207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1253/2499] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:12:38 2022] POST /identity/v3/users => generated 291 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:39.105973 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369b0ae8-7a01-4c0c-be1a-6b241cb3b679 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:39.122136 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369b0ae8-7a01-4c0c-be1a-6b241cb3b679 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:39.131954 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835373fc-8286-49d1-905d-d4900092c7db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:39.133399 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835373fc-8286-49d1-905d-d4900092c7db admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:39.133934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835373fc-8286-49d1-905d-d4900092c7db admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:39.134455 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835373fc-8286-49d1-905d-d4900092c7db admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:39.135530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835373fc-8286-49d1-905d-d4900092c7db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:39.137456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835373fc-8286-49d1-905d-d4900092c7db admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:39.161907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1247/2500] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:39 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:39.163185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72088218-6645-49a7-8ab2-e58574df5e41 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:39.173743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72088218-6645-49a7-8ab2-e58574df5e41 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:39.181910 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35436c6-d724-4b5e-98c6-c54d4e09f025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:39.182517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35436c6-d724-4b5e-98c6-c54d4e09f025 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:39.182793 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35436c6-d724-4b5e-98c6-c54d4e09f025 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:39.183061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35436c6-d724-4b5e-98c6-c54d4e09f025 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:39.183711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35436c6-d724-4b5e-98c6-c54d4e09f025 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:39.185375 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35436c6-d724-4b5e-98c6-c54d4e09f025 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:39.198339 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1254/2501] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:39 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:12:39.202580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835373fc-8286-49d1-905d-d4900092c7db admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:39.222230 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835373fc-8286-49d1-905d-d4900092c7db admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:39.228420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfcbd5f-9b56-4f26-879d-5eb5d93994e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:39.229373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfcbd5f-9b56-4f26-879d-5eb5d93994e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:39.229946 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfcbd5f-9b56-4f26-879d-5eb5d93994e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:39.230398 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfcbd5f-9b56-4f26-879d-5eb5d93994e6 admin admin] PATH_INFO: `/v3/projects/e67874d0fb224e80a4fb66434a4c2db8/users/196935b9920440e18cb629248fe56186/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:39.251678 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfcbd5f-9b56-4f26-879d-5eb5d93994e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e67874d0fb224e80a4fb66434a4c2db8, user_id=196935b9920440e18cb629248fe56186, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:39.254939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfcbd5f-9b56-4f26-879d-5eb5d93994e6 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:39.347970 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1248/2502] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:39 2022] PUT /identity/v3/projects/e67874d0fb224e80a4fb66434a4c2db8/users/196935b9920440e18cb629248fe56186/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:12:39.358508 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0094bd39-bd67-40b0-a06a-5201639c9613 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:39.358795 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0094bd39-bd67-40b0-a06a-5201639c9613 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:39.359073 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0094bd39-bd67-40b0-a06a-5201639c9613 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:39.396541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-0094bd39-bd67-40b0-a06a-5201639c9613 None None] MFA Rules not processed for user `196935b9920440e18cb629248fe56186`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:39.398333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0094bd39-bd67-40b0-a06a-5201639c9613 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:39.435145 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:39.488593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1255/2503] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:39 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:39.493467 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfcbd5f-9b56-4f26-879d-5eb5d93994e6 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:39.560885 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfcbd5f-9b56-4f26-879d-5eb5d93994e6 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:39.566689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a629fc-4f00-4bef-b197-5c8acfa95558 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:39.567350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a629fc-4f00-4bef-b197-5c8acfa95558 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:39.567629 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a629fc-4f00-4bef-b197-5c8acfa95558 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:39.567900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a629fc-4f00-4bef-b197-5c8acfa95558 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:39.568542 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a629fc-4f00-4bef-b197-5c8acfa95558 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:39.570481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a629fc-4f00-4bef-b197-5c8acfa95558 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:39.696537 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1249/2504] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:12:39 2022] POST /identity/v3/projects => generated 381 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:39.701116 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0094bd39-bd67-40b0-a06a-5201639c9613 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:39.713938 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a629fc-4f00-4bef-b197-5c8acfa95558 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:39.716128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-93a629fc-4f00-4bef-b197-5c8acfa95558 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:39.729070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0094bd39-bd67-40b0-a06a-5201639c9613 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:39.793206 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a629fc-4f00-4bef-b197-5c8acfa95558 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:39.797810 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1926d4cb-c72e-406f-907b-2c35d54958a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:39.798594 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1926d4cb-c72e-406f-907b-2c35d54958a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:39.798986 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1926d4cb-c72e-406f-907b-2c35d54958a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:39.799336 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1926d4cb-c72e-406f-907b-2c35d54958a0 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:39.800141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1926d4cb-c72e-406f-907b-2c35d54958a0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:39.800876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5308d8-8f92-41ad-9bf5-67a37c946ee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:39.801794 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5308d8-8f92-41ad-9bf5-67a37c946ee2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:39.802072 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5308d8-8f92-41ad-9bf5-67a37c946ee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:39.804156 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5308d8-8f92-41ad-9bf5-67a37c946ee2 admin admin] PATH_INFO: `/v3/users/f5e4723258c64ce5885154dded2d3d43` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:39.804272 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1926d4cb-c72e-406f-907b-2c35d54958a0 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:39.804381 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e5308d8-8f92-41ad-9bf5-67a37c946ee2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f5e4723258c64ce5885154dded2d3d43)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:39.805583 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e5308d8-8f92-41ad-9bf5-67a37c946ee2 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:39.901388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1256/2505] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:12:39 2022] POST /identity/v3/users => generated 297 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:39.905370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1926d4cb-c72e-406f-907b-2c35d54958a0 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:39.915176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1926d4cb-c72e-406f-907b-2c35d54958a0 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:39.919903 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f213203-a0d6-45fe-9c11-abf67cf5adf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:39.920476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f213203-a0d6-45fe-9c11-abf67cf5adf0 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:39.920753 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f213203-a0d6-45fe-9c11-abf67cf5adf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:39.921023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f213203-a0d6-45fe-9c11-abf67cf5adf0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:39.921726 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f213203-a0d6-45fe-9c11-abf67cf5adf0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:39.924006 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f213203-a0d6-45fe-9c11-abf67cf5adf0 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:39.933892 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1257/2506] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:39 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:12:39.937809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f213203-a0d6-45fe-9c11-abf67cf5adf0 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:39.947738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f213203-a0d6-45fe-9c11-abf67cf5adf0 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:39.952407 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0861a18a-b4d8-4d07-b608-3629681aa2a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:39.952695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=91,keystone:INSERT=3,keystone:DELETE=6 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:39.958061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0861a18a-b4d8-4d07-b608-3629681aa2a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:39.958061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0861a18a-b4d8-4d07-b608-3629681aa2a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:39.958061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0861a18a-b4d8-4d07-b608-3629681aa2a7 admin admin] PATH_INFO: `/v3/projects/77465baf5a44483d9df1d4c901c0cfd4/users/713b745e3d89439db81b96fda7066c64/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:39.958355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-2e5308d8-8f92-41ad-9bf5-67a37c946ee2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f5e4723258c64ce5885154dded2d3d43'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:39.958355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-2e5308d8-8f92-41ad-9bf5-67a37c946ee2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f5e4723258c64ce5885154dded2d3d43'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:39.968362 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0861a18a-b4d8-4d07-b608-3629681aa2a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=77465baf5a44483d9df1d4c901c0cfd4, user_id=713b745e3d89439db81b96fda7066c64, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:39.971348 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0861a18a-b4d8-4d07-b608-3629681aa2a7 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:40.003107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-2e5308d8-8f92-41ad-9bf5-67a37c946ee2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f5e4723258c64ce5885154dded2d3d43'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:40.015437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-2e5308d8-8f92-41ad-9bf5-67a37c946ee2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f5e4723258c64ce5885154dded2d3d43'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:40.016145 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-2e5308d8-8f92-41ad-9bf5-67a37c946ee2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f5e4723258c64ce5885154dded2d3d43'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:40.026176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1258/2507] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:39 2022] PUT /identity/v3/projects/77465baf5a44483d9df1d4c901c0cfd4/users/713b745e3d89439db81b96fda7066c64/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:12:40.029225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1250/2508] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:39 2022] DELETE /identity/v3/users/f5e4723258c64ce5885154dded2d3d43 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:40.030175 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0861a18a-b4d8-4d07-b608-3629681aa2a7 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:40.036876 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0861a18a-b4d8-4d07-b608-3629681aa2a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:40.103770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0861a18a-b4d8-4d07-b608-3629681aa2a7 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:40.134997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:40.163023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc49371-18e4-44cf-a81f-93a2e21c5917 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:40.163642 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc49371-18e4-44cf-a81f-93a2e21c5917 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:40.163919 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc49371-18e4-44cf-a81f-93a2e21c5917 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:40.164189 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc49371-18e4-44cf-a81f-93a2e21c5917 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:40.164834 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc49371-18e4-44cf-a81f-93a2e21c5917 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:40.166482 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc49371-18e4-44cf-a81f-93a2e21c5917 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:40.180612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1259/2509] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:40 2022] GET /identity/v3/roles => generated 1420 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:12:40.181813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5308d8-8f92-41ad-9bf5-67a37c946ee2 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:40.193124 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5308d8-8f92-41ad-9bf5-67a37c946ee2 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:40.197918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f250f0e2-5deb-42f2-8031-cd3439554cf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:40.198616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f250f0e2-5deb-42f2-8031-cd3439554cf7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:40.198900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f250f0e2-5deb-42f2-8031-cd3439554cf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:40.199175 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f250f0e2-5deb-42f2-8031-cd3439554cf7 admin admin] PATH_INFO: `/v3/projects/77465baf5a44483d9df1d4c901c0cfd4/users/713b745e3d89439db81b96fda7066c64/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:40.201319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f250f0e2-5deb-42f2-8031-cd3439554cf7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=77465baf5a44483d9df1d4c901c0cfd4, user_id=713b745e3d89439db81b96fda7066c64, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:40.203929 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f250f0e2-5deb-42f2-8031-cd3439554cf7 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:40.215017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1251/2510] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:40 2022] PUT /identity/v3/projects/77465baf5a44483d9df1d4c901c0cfd4/users/713b745e3d89439db81b96fda7066c64/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:40.226697 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff6877e-428d-427b-8039-93a6eba675b2 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:40.226988 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff6877e-428d-427b-8039-93a6eba675b2 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:40.227266 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff6877e-428d-427b-8039-93a6eba675b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:40.256769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f250f0e2-5deb-42f2-8031-cd3439554cf7 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:40.259168 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f250f0e2-5deb-42f2-8031-cd3439554cf7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:40.259300 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-6ff6877e-428d-427b-8039-93a6eba675b2 None None] MFA Rules not processed for user `713b745e3d89439db81b96fda7066c64`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:40.260481 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-6ff6877e-428d-427b-8039-93a6eba675b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:40.331488 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f250f0e2-5deb-42f2-8031-cd3439554cf7 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:40.336541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65e5e3f-3f0e-43b4-91c8-b4c4cb117cda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:40.336825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1260/2511] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:40 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:12:40.337348 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65e5e3f-3f0e-43b4-91c8-b4c4cb117cda admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:40.337689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65e5e3f-3f0e-43b4-91c8-b4c4cb117cda admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:40.338074 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65e5e3f-3f0e-43b4-91c8-b4c4cb117cda admin admin] PATH_INFO: `/v3/projects/287d111f08ac4b05bd5adec9c40d0620` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:40.339446 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65e5e3f-3f0e-43b4-91c8-b4c4cb117cda admin admin] RBAC: Authorizing `identity:delete_project(project_id=287d111f08ac4b05bd5adec9c40d0620)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:40.341608 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65e5e3f-3f0e-43b4-91c8-b4c4cb117cda admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:40.348089 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff6877e-428d-427b-8039-93a6eba675b2 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:40.349496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-6ff6877e-428d-427b-8039-93a6eba675b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:40.406116 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff6877e-428d-427b-8039-93a6eba675b2 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:40.410647 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6830d37d-b14d-40c3-921e-c082a89391af service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:40.411259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6830d37d-b14d-40c3-921e-c082a89391af service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:40.411567 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6830d37d-b14d-40c3-921e-c082a89391af service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:40.413868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6830d37d-b14d-40c3-921e-c082a89391af service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:40.421095 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6830d37d-b14d-40c3-921e-c082a89391af service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:40.422534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6830d37d-b14d-40c3-921e-c082a89391af service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:40.433197 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1261/2512] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:40 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:12:40.450065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c65e5e3f-3f0e-43b4-91c8-b4c4cb117cda admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '287d111f08ac4b05bd5adec9c40d0620', 'request_id': 'req-c65e5e3f-3f0e-43b4-91c8-b4c4cb117cda'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:40.455996 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c65e5e3f-3f0e-43b4-91c8-b4c4cb117cda admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '287d111f08ac4b05bd5adec9c40d0620', 'request_id': 'req-c65e5e3f-3f0e-43b4-91c8-b4c4cb117cda'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:40.535740 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c65e5e3f-3f0e-43b4-91c8-b4c4cb117cda admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '287d111f08ac4b05bd5adec9c40d0620', 'request_id': 'req-c65e5e3f-3f0e-43b4-91c8-b4c4cb117cda'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:40.548716 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6830d37d-b14d-40c3-921e-c082a89391af service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:40.552796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c65e5e3f-3f0e-43b4-91c8-b4c4cb117cda admin admin] The token cache is being invalidate because project 287d111f08ac4b05bd5adec9c40d0620 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:12:40.553192 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c65e5e3f-3f0e-43b4-91c8-b4c4cb117cda admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:40.553939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-c65e5e3f-3f0e-43b4-91c8-b4c4cb117cda admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:40.555886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6830d37d-b14d-40c3-921e-c082a89391af service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:40.559232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1252/2513] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:12:40 2022] DELETE /identity/v3/projects/287d111f08ac4b05bd5adec9c40d0620 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:12:40.560561 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd614327-60df-4794-95e4-e6b9eead4a1d None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:40.560846 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd614327-60df-4794-95e4-e6b9eead4a1d None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:40.561148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd614327-60df-4794-95e4-e6b9eead4a1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:40.584233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-cd614327-60df-4794-95e4-e6b9eead4a1d None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:40.585515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-cd614327-60df-4794-95e4-e6b9eead4a1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:40.616886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438704b8-6756-4950-ae6c-759157f9c186 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:40.617527 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438704b8-6756-4950-ae6c-759157f9c186 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:40.617812 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438704b8-6756-4950-ae6c-759157f9c186 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:40.618085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438704b8-6756-4950-ae6c-759157f9c186 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:40.619061 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:DELETE=11,keystone:SELECT=86,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:40.619932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-438704b8-6756-4950-ae6c-759157f9c186 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:40.630076 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:40.645655 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1253/2514] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:40.653481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39f0ca8-dcfe-4e3c-85ca-147288d1bfc3 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:40.658098 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39f0ca8-dcfe-4e3c-85ca-147288d1bfc3 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:40.658098 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39f0ca8-dcfe-4e3c-85ca-147288d1bfc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:40.674069 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-d39f0ca8-dcfe-4e3c-85ca-147288d1bfc3 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:40.675255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d39f0ca8-dcfe-4e3c-85ca-147288d1bfc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:40.682972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438704b8-6756-4950-ae6c-759157f9c186 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:40.686549 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1254/2515] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:12:40.686659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438704b8-6756-4950-ae6c-759157f9c186 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:40.687419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39f0ca8-dcfe-4e3c-85ca-147288d1bfc3 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:40.694124 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1262/2516] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:40 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:12:40.698069 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39f0ca8-dcfe-4e3c-85ca-147288d1bfc3 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:40.702576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad50d19-403a-41aa-81bd-8942f66abf9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:40.703161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad50d19-403a-41aa-81bd-8942f66abf9f admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:40.703437 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad50d19-403a-41aa-81bd-8942f66abf9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:40.703707 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad50d19-403a-41aa-81bd-8942f66abf9f admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:40.705040 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad50d19-403a-41aa-81bd-8942f66abf9f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:12:40.705386 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad50d19-403a-41aa-81bd-8942f66abf9f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:40.708085 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad50d19-403a-41aa-81bd-8942f66abf9f admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:40.720630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1255/2517] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:12:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 01:12:40.725210 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438704b8-6756-4950-ae6c-759157f9c186 service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:40.736335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438704b8-6756-4950-ae6c-759157f9c186 service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:40.741021 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87bf1c7f-7c74-415a-a119-2837d355c0a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:40.741997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bf1c7f-7c74-415a-a119-2837d355c0a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:40.742345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bf1c7f-7c74-415a-a119-2837d355c0a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:40.742689 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87bf1c7f-7c74-415a-a119-2837d355c0a4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:40.743388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87bf1c7f-7c74-415a-a119-2837d355c0a4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:40.745486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87bf1c7f-7c74-415a-a119-2837d355c0a4 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:40.821533 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1263/2518] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:12:40 2022] POST /identity/v3/projects => generated 365 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:40.826047 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad50d19-403a-41aa-81bd-8942f66abf9f admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:40.836840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad50d19-403a-41aa-81bd-8942f66abf9f admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:40.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66574e2d-ac87-453d-b1ad-b4395c76bfb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:40.895982 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66574e2d-ac87-453d-b1ad-b4395c76bfb7 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:40.896262 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66574e2d-ac87-453d-b1ad-b4395c76bfb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:40.896533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66574e2d-ac87-453d-b1ad-b4395c76bfb7 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:40.897210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66574e2d-ac87-453d-b1ad-b4395c76bfb7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:40.899131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66574e2d-ac87-453d-b1ad-b4395c76bfb7 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:40.998424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1256/2519] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:12:40 2022] POST /identity/v3/users => generated 283 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:41.003639 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87bf1c7f-7c74-415a-a119-2837d355c0a4 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:41.021530 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87bf1c7f-7c74-415a-a119-2837d355c0a4 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:41.026371 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3889ba31-d0f5-4c4f-93e7-d2c434710f51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:41.027042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3889ba31-d0f5-4c4f-93e7-d2c434710f51 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:41.027328 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3889ba31-d0f5-4c4f-93e7-d2c434710f51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:41.027597 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3889ba31-d0f5-4c4f-93e7-d2c434710f51 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:41.028243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3889ba31-d0f5-4c4f-93e7-d2c434710f51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:41.030023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3889ba31-d0f5-4c4f-93e7-d2c434710f51 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:41.042757 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1264/2520] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:41 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:12:41.046948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66574e2d-ac87-453d-b1ad-b4395c76bfb7 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:41.057030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66574e2d-ac87-453d-b1ad-b4395c76bfb7 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:41.061850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d317b365-70e3-484e-beb6-aac47543bdf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:41.062454 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d317b365-70e3-484e-beb6-aac47543bdf2 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:41.062736 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d317b365-70e3-484e-beb6-aac47543bdf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:41.063009 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d317b365-70e3-484e-beb6-aac47543bdf2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:41.064176 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d317b365-70e3-484e-beb6-aac47543bdf2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:41.065802 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d317b365-70e3-484e-beb6-aac47543bdf2 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:41.077419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1257/2521] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:41.082682 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3889ba31-d0f5-4c4f-93e7-d2c434710f51 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:41.092899 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3889ba31-d0f5-4c4f-93e7-d2c434710f51 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:41.098176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d18401-8e90-49e8-bb4c-82846111b7ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:41.101441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d18401-8e90-49e8-bb4c-82846111b7ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:41.101441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d18401-8e90-49e8-bb4c-82846111b7ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:41.101441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d18401-8e90-49e8-bb4c-82846111b7ff admin admin] PATH_INFO: `/v3/projects/0a46e1b837a14345b3d11969f64f3dae/users/4bcf4aeb82bc4ad7b91a8fdb7f23bdfe/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:41.114991 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d18401-8e90-49e8-bb4c-82846111b7ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a46e1b837a14345b3d11969f64f3dae, user_id=4bcf4aeb82bc4ad7b91a8fdb7f23bdfe, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:41.117984 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d18401-8e90-49e8-bb4c-82846111b7ff admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:41.209455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1265/2522] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:41 2022] PUT /identity/v3/projects/0a46e1b837a14345b3d11969f64f3dae/users/4bcf4aeb82bc4ad7b91a8fdb7f23bdfe/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:12:41.220827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a571a3fe-39f8-4ac9-a9cd-71abecbbd2eb None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:41.221138 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a571a3fe-39f8-4ac9-a9cd-71abecbbd2eb None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:41.221422 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a571a3fe-39f8-4ac9-a9cd-71abecbbd2eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:41.249556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-a571a3fe-39f8-4ac9-a9cd-71abecbbd2eb None None] MFA Rules not processed for user `4bcf4aeb82bc4ad7b91a8fdb7f23bdfe`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:41.250866 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a571a3fe-39f8-4ac9-a9cd-71abecbbd2eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:41.321617 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1258/2523] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:41 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:41.326034 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d18401-8e90-49e8-bb4c-82846111b7ff admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:41.397993 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d18401-8e90-49e8-bb4c-82846111b7ff admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:41.403486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ebc546-b796-425f-8feb-1c3786b30b30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:41.404203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ebc546-b796-425f-8feb-1c3786b30b30 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:41.404604 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ebc546-b796-425f-8feb-1c3786b30b30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:41.404876 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ebc546-b796-425f-8feb-1c3786b30b30 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:41.406157 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ebc546-b796-425f-8feb-1c3786b30b30 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:41.407666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ebc546-b796-425f-8feb-1c3786b30b30 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:41.524766 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1266/2524] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:12:41 2022] POST /identity/v3/projects => generated 363 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:12:41.528987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a571a3fe-39f8-4ac9-a9cd-71abecbbd2eb None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:41.539541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a571a3fe-39f8-4ac9-a9cd-71abecbbd2eb None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:41.565772 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:41.597730 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9406f98-d793-4e44-93b4-51775922993c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:41.598354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9406f98-d793-4e44-93b4-51775922993c admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:41.598627 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9406f98-d793-4e44-93b4-51775922993c admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:41.610170 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9406f98-d793-4e44-93b4-51775922993c admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:41.610170 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9406f98-d793-4e44-93b4-51775922993c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:41.610170 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9406f98-d793-4e44-93b4-51775922993c admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:41.689873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1259/2525] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:12:41 2022] POST /identity/v3/users => generated 288 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:41.693837 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ebc546-b796-425f-8feb-1c3786b30b30 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:41.704288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ebc546-b796-425f-8feb-1c3786b30b30 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:41.709219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2efbcd9-b741-40c1-a9da-503978ebe718 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:41.709845 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2efbcd9-b741-40c1-a9da-503978ebe718 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:41.710201 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2efbcd9-b741-40c1-a9da-503978ebe718 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:41.710478 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2efbcd9-b741-40c1-a9da-503978ebe718 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:41.711146 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2efbcd9-b741-40c1-a9da-503978ebe718 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:41.712778 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2efbcd9-b741-40c1-a9da-503978ebe718 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:41.724430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1267/2526] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:12:41.728346 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9406f98-d793-4e44-93b4-51775922993c admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:41.739655 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9406f98-d793-4e44-93b4-51775922993c admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:41.744518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e74ef63-fa8e-4d1b-84f6-b9ed88828f66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:41.745282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e74ef63-fa8e-4d1b-84f6-b9ed88828f66 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:41.745561 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e74ef63-fa8e-4d1b-84f6-b9ed88828f66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:41.745832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e74ef63-fa8e-4d1b-84f6-b9ed88828f66 admin admin] PATH_INFO: `/v3/projects/fdb2b7eabf214f66a355bd53eebdb9f5/users/3a2e61adfbf74968a5a126e313ef461c/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:41.762805 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e74ef63-fa8e-4d1b-84f6-b9ed88828f66 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fdb2b7eabf214f66a355bd53eebdb9f5, user_id=3a2e61adfbf74968a5a126e313ef461c, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:41.771749 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e74ef63-fa8e-4d1b-84f6-b9ed88828f66 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:41.832172 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1260/2527] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:41 2022] PUT /identity/v3/projects/fdb2b7eabf214f66a355bd53eebdb9f5/users/3a2e61adfbf74968a5a126e313ef461c/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:12:41.841669 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2efbcd9-b741-40c1-a9da-503978ebe718 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:41.852400 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2efbcd9-b741-40c1-a9da-503978ebe718 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:41.878163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:41.912023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b58eec7-8929-4833-839c-b1b7c4cf72f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:41.912624 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b58eec7-8929-4833-839c-b1b7c4cf72f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:41.912901 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b58eec7-8929-4833-839c-b1b7c4cf72f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:41.913204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b58eec7-8929-4833-839c-b1b7c4cf72f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:41.913841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b58eec7-8929-4833-839c-b1b7c4cf72f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:41.915422 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b58eec7-8929-4833-839c-b1b7c4cf72f1 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:41.929337 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1268/2528] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:41 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:41.930727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e74ef63-fa8e-4d1b-84f6-b9ed88828f66 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:41.941766 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e74ef63-fa8e-4d1b-84f6-b9ed88828f66 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:41.946583 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d4b8b5-56e6-4c56-8b85-4d3ef3f634b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:41.947380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d4b8b5-56e6-4c56-8b85-4d3ef3f634b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:41.947659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d4b8b5-56e6-4c56-8b85-4d3ef3f634b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:41.947996 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d4b8b5-56e6-4c56-8b85-4d3ef3f634b9 admin admin] PATH_INFO: `/v3/projects/fdb2b7eabf214f66a355bd53eebdb9f5/users/3a2e61adfbf74968a5a126e313ef461c/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:41.950214 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d4b8b5-56e6-4c56-8b85-4d3ef3f634b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fdb2b7eabf214f66a355bd53eebdb9f5, user_id=3a2e61adfbf74968a5a126e313ef461c, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:41.953036 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d4b8b5-56e6-4c56-8b85-4d3ef3f634b9 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:41.975887 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1261/2529] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:41 2022] PUT /identity/v3/projects/fdb2b7eabf214f66a355bd53eebdb9f5/users/3a2e61adfbf74968a5a126e313ef461c/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:41.987772 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eada839f-809a-4120-8b46-d7dc0a1fe490 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:41.988146 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eada839f-809a-4120-8b46-d7dc0a1fe490 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:41.988475 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eada839f-809a-4120-8b46-d7dc0a1fe490 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:42.019362 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-eada839f-809a-4120-8b46-d7dc0a1fe490 None None] MFA Rules not processed for user `3a2e61adfbf74968a5a126e313ef461c`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:42.020591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-eada839f-809a-4120-8b46-d7dc0a1fe490 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:42.100002 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1269/2530] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:41 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:42.112759 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d4b8b5-56e6-4c56-8b85-4d3ef3f634b9 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:42.114576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e7d4b8b5-56e6-4c56-8b85-4d3ef3f634b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:42.183516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d4b8b5-56e6-4c56-8b85-4d3ef3f634b9 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:42.190023 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a14e88-0487-4421-ad91-b702fcba772e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:42.195072 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a14e88-0487-4421-ad91-b702fcba772e service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:42.195072 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a14e88-0487-4421-ad91-b702fcba772e service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:42.195072 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a14e88-0487-4421-ad91-b702fcba772e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:42.208004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a14e88-0487-4421-ad91-b702fcba772e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:42.210033 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a14e88-0487-4421-ad91-b702fcba772e service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:42.228611 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1262/2531] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:42 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:42.525273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eada839f-809a-4120-8b46-d7dc0a1fe490 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:42.531825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-eada839f-809a-4120-8b46-d7dc0a1fe490 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:42.539226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eada839f-809a-4120-8b46-d7dc0a1fe490 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:42.542890 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e6fb89-8fe2-4364-8326-91933cf8c4e4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:42.543493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e6fb89-8fe2-4364-8326-91933cf8c4e4 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:42.543771 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e6fb89-8fe2-4364-8326-91933cf8c4e4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:42.544047 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e6fb89-8fe2-4364-8326-91933cf8c4e4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:42.612284 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e6fb89-8fe2-4364-8326-91933cf8c4e4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:42.613881 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e6fb89-8fe2-4364-8326-91933cf8c4e4 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:42.625106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1270/2532] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:42 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:43.857824 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a14e88-0487-4421-ad91-b702fcba772e service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:43.859232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a5a14e88-0487-4421-ad91-b702fcba772e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:43.924915 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a14e88-0487-4421-ad91-b702fcba772e service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:43.929543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2b2796-ef21-4772-b3f7-763abdacdbb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:43.930168 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2b2796-ef21-4772-b3f7-763abdacdbb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:43.930452 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2b2796-ef21-4772-b3f7-763abdacdbb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:43.930727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2b2796-ef21-4772-b3f7-763abdacdbb3 admin admin] PATH_INFO: `/v3/users/196935b9920440e18cb629248fe56186` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:43.931849 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2b2796-ef21-4772-b3f7-763abdacdbb3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=196935b9920440e18cb629248fe56186)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:43.933874 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2b2796-ef21-4772-b3f7-763abdacdbb3 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:43.956566 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=91,keystone:INSERT=5,keystone:DELETE=4 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:44.133211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-be2b2796-ef21-4772-b3f7-763abdacdbb3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '196935b9920440e18cb629248fe56186'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:44.133876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-be2b2796-ef21-4772-b3f7-763abdacdbb3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '196935b9920440e18cb629248fe56186'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:44.231925 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-be2b2796-ef21-4772-b3f7-763abdacdbb3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '196935b9920440e18cb629248fe56186'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:44.246196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-be2b2796-ef21-4772-b3f7-763abdacdbb3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '196935b9920440e18cb629248fe56186'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:44.246941 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-be2b2796-ef21-4772-b3f7-763abdacdbb3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '196935b9920440e18cb629248fe56186'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:44.260262 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1263/2533] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:43 2022] DELETE /identity/v3/users/196935b9920440e18cb629248fe56186 => generated 0 bytes in 403 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:44.264884 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e533e51-584f-4511-b792-62da231d95ca None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:44.265404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e533e51-584f-4511-b792-62da231d95ca None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:44.265510 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e533e51-584f-4511-b792-62da231d95ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:44.285415 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-7e533e51-584f-4511-b792-62da231d95ca None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:44.286627 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7e533e51-584f-4511-b792-62da231d95ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:44.328250 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:44.346099 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1271/2534] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:44.355938 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2b2796-ef21-4772-b3f7-763abdacdbb3 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:44.357410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-be2b2796-ef21-4772-b3f7-763abdacdbb3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:44.425000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2b2796-ef21-4772-b3f7-763abdacdbb3 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:44.428616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788b4bbc-105c-4251-99ea-55923139e877 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:44.429272 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788b4bbc-105c-4251-99ea-55923139e877 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:44.429548 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788b4bbc-105c-4251-99ea-55923139e877 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:44.429815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788b4bbc-105c-4251-99ea-55923139e877 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:44.440058 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788b4bbc-105c-4251-99ea-55923139e877 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:44.441570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788b4bbc-105c-4251-99ea-55923139e877 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:44.451174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1264/2535] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:45.183036 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e533e51-584f-4511-b792-62da231d95ca None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:45.185384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7e533e51-584f-4511-b792-62da231d95ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:45.201049 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e533e51-584f-4511-b792-62da231d95ca None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:45.206743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:45.207706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:45.208059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:45.208933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14 admin admin] PATH_INFO: `/v3/projects/e67874d0fb224e80a4fb66434a4c2db8` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:45.210344 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e67874d0fb224e80a4fb66434a4c2db8)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:45.213564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:45.355365 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e67874d0fb224e80a4fb66434a4c2db8', 'request_id': 'req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:45.361487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e67874d0fb224e80a4fb66434a4c2db8', 'request_id': 'req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:45.368391 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e67874d0fb224e80a4fb66434a4c2db8', 'request_id': 'req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:45.436956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14 admin admin] The token cache is being invalidate because project e67874d0fb224e80a4fb66434a4c2db8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:12:45.437278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:45.438003 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:45.439866 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1272/2536] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:12:45 2022] DELETE /identity/v3/projects/e67874d0fb224e80a4fb66434a4c2db8 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:12:45.444738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788b4bbc-105c-4251-99ea-55923139e877 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:45.455052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788b4bbc-105c-4251-99ea-55923139e877 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:45.515903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc3d2f4-a9fd-41c3-b519-d0fe0f072f66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:45.517172 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc3d2f4-a9fd-41c3-b519-d0fe0f072f66 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:45.517461 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc3d2f4-a9fd-41c3-b519-d0fe0f072f66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:45.517758 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc3d2f4-a9fd-41c3-b519-d0fe0f072f66 admin admin] PATH_INFO: `/v3/users/713b745e3d89439db81b96fda7066c64` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:45.518911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc3d2f4-a9fd-41c3-b519-d0fe0f072f66 admin admin] RBAC: Authorizing `identity:delete_user(user_id=713b745e3d89439db81b96fda7066c64)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:45.521095 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc3d2f4-a9fd-41c3-b519-d0fe0f072f66 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:45.671894 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:45.673570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ccc3d2f4-a9fd-41c3-b519-d0fe0f072f66 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '713b745e3d89439db81b96fda7066c64'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:45.674237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ccc3d2f4-a9fd-41c3-b519-d0fe0f072f66 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '713b745e3d89439db81b96fda7066c64'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:45.740626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ccc3d2f4-a9fd-41c3-b519-d0fe0f072f66 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '713b745e3d89439db81b96fda7066c64'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:45.756123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ccc3d2f4-a9fd-41c3-b519-d0fe0f072f66 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '713b745e3d89439db81b96fda7066c64'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:45.757298 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-ccc3d2f4-a9fd-41c3-b519-d0fe0f072f66 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '713b745e3d89439db81b96fda7066c64'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:45.772297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1265/2537] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:45 2022] DELETE /identity/v3/users/713b745e3d89439db81b96fda7066c64 => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:46.170030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:46.171569 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:46.186695 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deeb684a-3ce3-4ca9-b41e-5c860e50cc14 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:46.262476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783b11dc-236c-4669-9aa6-75fbd23c10bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:46.263127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783b11dc-236c-4669-9aa6-75fbd23c10bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:46.266277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783b11dc-236c-4669-9aa6-75fbd23c10bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:46.266277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783b11dc-236c-4669-9aa6-75fbd23c10bc admin admin] PATH_INFO: `/v3/projects/77465baf5a44483d9df1d4c901c0cfd4` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:46.266277 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783b11dc-236c-4669-9aa6-75fbd23c10bc admin admin] RBAC: Authorizing `identity:delete_project(project_id=77465baf5a44483d9df1d4c901c0cfd4)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:46.266789 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783b11dc-236c-4669-9aa6-75fbd23c10bc admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:46.352019 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-783b11dc-236c-4669-9aa6-75fbd23c10bc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '77465baf5a44483d9df1d4c901c0cfd4', 'request_id': 'req-783b11dc-236c-4669-9aa6-75fbd23c10bc'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:46.357407 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-783b11dc-236c-4669-9aa6-75fbd23c10bc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '77465baf5a44483d9df1d4c901c0cfd4', 'request_id': 'req-783b11dc-236c-4669-9aa6-75fbd23c10bc'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:46.364373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-783b11dc-236c-4669-9aa6-75fbd23c10bc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '77465baf5a44483d9df1d4c901c0cfd4', 'request_id': 'req-783b11dc-236c-4669-9aa6-75fbd23c10bc'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:46.411154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:46.415135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-783b11dc-236c-4669-9aa6-75fbd23c10bc admin admin] The token cache is being invalidate because project 77465baf5a44483d9df1d4c901c0cfd4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:12:46.415541 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-783b11dc-236c-4669-9aa6-75fbd23c10bc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:46.416347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-783b11dc-236c-4669-9aa6-75fbd23c10bc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:46.417924 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1273/2538] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:12:46 2022] DELETE /identity/v3/projects/77465baf5a44483d9df1d4c901c0cfd4 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:46.422994 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234e8bb1-1e83-4a2c-b402-d8af16930faf None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:46.423398 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234e8bb1-1e83-4a2c-b402-d8af16930faf None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:46.423702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234e8bb1-1e83-4a2c-b402-d8af16930faf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:46.453154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-234e8bb1-1e83-4a2c-b402-d8af16930faf None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:46.454508 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-234e8bb1-1e83-4a2c-b402-d8af16930faf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:46.521234 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1266/2539] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:46.531670 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea7145e-8083-4d53-b72e-cc4f64f3d06c None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:46.531670 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea7145e-8083-4d53-b72e-cc4f64f3d06c None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:46.531670 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea7145e-8083-4d53-b72e-cc4f64f3d06c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:46.552869 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-8ea7145e-8083-4d53-b72e-cc4f64f3d06c None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:46.554881 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-8ea7145e-8083-4d53-b72e-cc4f64f3d06c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:46.561983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1274/2540] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:46.566456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234e8bb1-1e83-4a2c-b402-d8af16930faf None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:46.576057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234e8bb1-1e83-4a2c-b402-d8af16930faf None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:46.580804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d624cd-fca0-4e96-91cc-40a051ae79d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:46.581519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d624cd-fca0-4e96-91cc-40a051ae79d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:46.581794 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d624cd-fca0-4e96-91cc-40a051ae79d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:46.582066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d624cd-fca0-4e96-91cc-40a051ae79d8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:46.582746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d624cd-fca0-4e96-91cc-40a051ae79d8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:12:46.583053 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d624cd-fca0-4e96-91cc-40a051ae79d8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:46.584647 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d624cd-fca0-4e96-91cc-40a051ae79d8 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:46.596519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1267/2541] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:12:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:12:46.600812 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea7145e-8083-4d53-b72e-cc4f64f3d06c None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:46.610101 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea7145e-8083-4d53-b72e-cc4f64f3d06c None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:46.615626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b34f1c-4371-4710-ad05-1a6233be05b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:46.618104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b34f1c-4371-4710-ad05-1a6233be05b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:46.618104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b34f1c-4371-4710-ad05-1a6233be05b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:46.618104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b34f1c-4371-4710-ad05-1a6233be05b3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:46.618104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b34f1c-4371-4710-ad05-1a6233be05b3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:46.619612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b34f1c-4371-4710-ad05-1a6233be05b3 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:46.707857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1275/2542] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:12:46 2022] POST /identity/v3/projects => generated 409 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:46.712253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d624cd-fca0-4e96-91cc-40a051ae79d8 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:46.723237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d624cd-fca0-4e96-91cc-40a051ae79d8 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:46.790084 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40a5fa8-8551-43f6-8478-42943e0ca4f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:46.790704 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40a5fa8-8551-43f6-8478-42943e0ca4f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:46.790987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40a5fa8-8551-43f6-8478-42943e0ca4f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:46.791257 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40a5fa8-8551-43f6-8478-42943e0ca4f9 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:46.791911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40a5fa8-8551-43f6-8478-42943e0ca4f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:46.794216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40a5fa8-8551-43f6-8478-42943e0ca4f9 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:46.870860 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1268/2543] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:12:46 2022] POST /identity/v3/users => generated 305 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:46.872415 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b34f1c-4371-4710-ad05-1a6233be05b3 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:46.888181 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b34f1c-4371-4710-ad05-1a6233be05b3 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:46.893965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d7f826-945b-4274-88ce-9cfacd94423e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:46.894734 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d7f826-945b-4274-88ce-9cfacd94423e admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:46.895019 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d7f826-945b-4274-88ce-9cfacd94423e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:46.895292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d7f826-945b-4274-88ce-9cfacd94423e admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:46.896078 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d7f826-945b-4274-88ce-9cfacd94423e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:46.897715 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d7f826-945b-4274-88ce-9cfacd94423e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:46.913739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1276/2544] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:46 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:46.915384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40a5fa8-8551-43f6-8478-42943e0ca4f9 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:46.925805 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40a5fa8-8551-43f6-8478-42943e0ca4f9 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:46.930653 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ae857e-45c0-4105-909a-381f21c71753 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:46.931249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ae857e-45c0-4105-909a-381f21c71753 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:46.931530 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ae857e-45c0-4105-909a-381f21c71753 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:46.931803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ae857e-45c0-4105-909a-381f21c71753 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:46.932467 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ae857e-45c0-4105-909a-381f21c71753 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:46.934126 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30ae857e-45c0-4105-909a-381f21c71753 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:46.945181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1269/2545] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:46.949505 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d7f826-945b-4274-88ce-9cfacd94423e admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:46.960457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d7f826-945b-4274-88ce-9cfacd94423e admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:46.965677 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f35816-1295-4bf6-a0e0-48061330946e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:46.966318 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f35816-1295-4bf6-a0e0-48061330946e admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:46.966597 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f35816-1295-4bf6-a0e0-48061330946e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:46.966872 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f35816-1295-4bf6-a0e0-48061330946e admin admin] PATH_INFO: `/v3/projects/0af4373504224763a31adfd64271d0a2/users/a02db93f4e5d4255ac3c8f6d4729df35/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:46.982264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f35816-1295-4bf6-a0e0-48061330946e admin admin] RBAC: Authorizing `identity:create_grant(project_id=0af4373504224763a31adfd64271d0a2, user_id=a02db93f4e5d4255ac3c8f6d4729df35, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:46.985144 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f35816-1295-4bf6-a0e0-48061330946e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:47.033590 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1277/2546] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:46 2022] PUT /identity/v3/projects/0af4373504224763a31adfd64271d0a2/users/a02db93f4e5d4255ac3c8f6d4729df35/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:12:47.048175 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918c6a65-d4aa-49b8-a8bc-b0c2a90b7273 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:47.048462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918c6a65-d4aa-49b8-a8bc-b0c2a90b7273 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:47.048741 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918c6a65-d4aa-49b8-a8bc-b0c2a90b7273 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:47.084296 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-918c6a65-d4aa-49b8-a8bc-b0c2a90b7273 None None] MFA Rules not processed for user `a02db93f4e5d4255ac3c8f6d4729df35`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:47.085578 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-918c6a65-d4aa-49b8-a8bc-b0c2a90b7273 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:47.108685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:47.155910 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1270/2547] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:47 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:47.160638 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f35816-1295-4bf6-a0e0-48061330946e admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:47.222107 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f35816-1295-4bf6-a0e0-48061330946e admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:47.226780 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1874354-eb7d-4ad7-8959-16b5d28ae544 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:47.227377 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1874354-eb7d-4ad7-8959-16b5d28ae544 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:47.227658 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1874354-eb7d-4ad7-8959-16b5d28ae544 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:47.227929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1874354-eb7d-4ad7-8959-16b5d28ae544 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:47.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1874354-eb7d-4ad7-8959-16b5d28ae544 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:47.230506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1874354-eb7d-4ad7-8959-16b5d28ae544 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:47.335537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1278/2548] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:12:47 2022] POST /identity/v3/projects => generated 409 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:47.339608 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918c6a65-d4aa-49b8-a8bc-b0c2a90b7273 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:47.350519 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918c6a65-d4aa-49b8-a8bc-b0c2a90b7273 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:47.406514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427cdb8c-40f6-4637-8621-876597876f01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:47.407117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427cdb8c-40f6-4637-8621-876597876f01 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:47.407399 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427cdb8c-40f6-4637-8621-876597876f01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:47.407672 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427cdb8c-40f6-4637-8621-876597876f01 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:47.408325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427cdb8c-40f6-4637-8621-876597876f01 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:47.410309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427cdb8c-40f6-4637-8621-876597876f01 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:47.489520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1271/2549] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:12:47 2022] POST /identity/v3/users => generated 311 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:47.493927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1874354-eb7d-4ad7-8959-16b5d28ae544 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:47.504959 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1874354-eb7d-4ad7-8959-16b5d28ae544 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:47.509888 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55863b06-fdcd-4abe-9d0f-44c71b0ca2b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:47.510476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55863b06-fdcd-4abe-9d0f-44c71b0ca2b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:47.510752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55863b06-fdcd-4abe-9d0f-44c71b0ca2b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:47.511020 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55863b06-fdcd-4abe-9d0f-44c71b0ca2b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:47.511654 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55863b06-fdcd-4abe-9d0f-44c71b0ca2b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:47.513315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55863b06-fdcd-4abe-9d0f-44c71b0ca2b2 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:47.523955 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1279/2550] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:47 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:47.528053 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427cdb8c-40f6-4637-8621-876597876f01 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:47.538294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427cdb8c-40f6-4637-8621-876597876f01 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:47.542931 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fc7fff-579c-40c6-99f5-513b71b0d442 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:47.543604 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fc7fff-579c-40c6-99f5-513b71b0d442 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:47.543884 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fc7fff-579c-40c6-99f5-513b71b0d442 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:47.544158 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fc7fff-579c-40c6-99f5-513b71b0d442 admin admin] PATH_INFO: `/v3/projects/15ed0cea1cd345429921b1083d321708/users/1cd350479e314e069257c2c19e019a98/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:47.559685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fc7fff-579c-40c6-99f5-513b71b0d442 admin admin] RBAC: Authorizing `identity:create_grant(project_id=15ed0cea1cd345429921b1083d321708, user_id=1cd350479e314e069257c2c19e019a98, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:47.562378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fc7fff-579c-40c6-99f5-513b71b0d442 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:47.625108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1272/2551] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:47 2022] PUT /identity/v3/projects/15ed0cea1cd345429921b1083d321708/users/1cd350479e314e069257c2c19e019a98/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:47.629642 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55863b06-fdcd-4abe-9d0f-44c71b0ca2b2 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:47.639796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55863b06-fdcd-4abe-9d0f-44c71b0ca2b2 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:47.696150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ca2c86-7875-45b5-bac1-548b10b46f6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:47.696745 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ca2c86-7875-45b5-bac1-548b10b46f6b admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:47.697021 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ca2c86-7875-45b5-bac1-548b10b46f6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:47.697325 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ca2c86-7875-45b5-bac1-548b10b46f6b admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:47.698066 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ca2c86-7875-45b5-bac1-548b10b46f6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:47.699652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ca2c86-7875-45b5-bac1-548b10b46f6b admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:47.709447 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1280/2552] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:47 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:47.713602 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fc7fff-579c-40c6-99f5-513b71b0d442 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:47.724404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fc7fff-579c-40c6-99f5-513b71b0d442 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:47.729012 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710b5ae0-e4ef-4210-a150-692645f92d97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:47.729711 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710b5ae0-e4ef-4210-a150-692645f92d97 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:47.729994 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710b5ae0-e4ef-4210-a150-692645f92d97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:47.730268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710b5ae0-e4ef-4210-a150-692645f92d97 admin admin] PATH_INFO: `/v3/projects/15ed0cea1cd345429921b1083d321708/users/1cd350479e314e069257c2c19e019a98/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:47.732314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710b5ae0-e4ef-4210-a150-692645f92d97 admin admin] RBAC: Authorizing `identity:create_grant(project_id=15ed0cea1cd345429921b1083d321708, user_id=1cd350479e314e069257c2c19e019a98, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:47.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710b5ae0-e4ef-4210-a150-692645f92d97 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:47.746361 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1273/2553] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:47 2022] PUT /identity/v3/projects/15ed0cea1cd345429921b1083d321708/users/1cd350479e314e069257c2c19e019a98/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:47.756450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8346390-0d0b-4bab-965d-2b02e5debb33 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:47.756738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8346390-0d0b-4bab-965d-2b02e5debb33 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:47.757016 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8346390-0d0b-4bab-965d-2b02e5debb33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:47.770423 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:47.785252 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-a8346390-0d0b-4bab-965d-2b02e5debb33 None None] MFA Rules not processed for user `1cd350479e314e069257c2c19e019a98`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:47.786576 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a8346390-0d0b-4bab-965d-2b02e5debb33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:47.854075 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1281/2554] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:47 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:47.865382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710b5ae0-e4ef-4210-a150-692645f92d97 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:47.867326 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-710b5ae0-e4ef-4210-a150-692645f92d97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:47.924205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710b5ae0-e4ef-4210-a150-692645f92d97 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:47.928500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80759fbe-e4d5-4936-a358-e7cfc505a9cb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:47.929125 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80759fbe-e4d5-4936-a358-e7cfc505a9cb service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:47.929430 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80759fbe-e4d5-4936-a358-e7cfc505a9cb service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:47.929819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80759fbe-e4d5-4936-a358-e7cfc505a9cb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:47.938991 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80759fbe-e4d5-4936-a358-e7cfc505a9cb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:47.940428 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80759fbe-e4d5-4936-a358-e7cfc505a9cb service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:47.950022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1274/2555] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:47 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:12:48.322072 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8346390-0d0b-4bab-965d-2b02e5debb33 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:48.332574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8346390-0d0b-4bab-965d-2b02e5debb33 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:48.336265 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4163d9-6b3f-4458-91f8-e6b36b92c891 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:48.336856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4163d9-6b3f-4458-91f8-e6b36b92c891 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:48.337161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4163d9-6b3f-4458-91f8-e6b36b92c891 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:48.337434 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4163d9-6b3f-4458-91f8-e6b36b92c891 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:48.393135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4163d9-6b3f-4458-91f8-e6b36b92c891 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:48.395238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4163d9-6b3f-4458-91f8-e6b36b92c891 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:48.404028 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1282/2556] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:48 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:51.372078 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80759fbe-e4d5-4936-a358-e7cfc505a9cb service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:51.373624 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-80759fbe-e4d5-4936-a358-e7cfc505a9cb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:51.387962 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:51.431802 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80759fbe-e4d5-4936-a358-e7cfc505a9cb service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:51.435486 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c97ca87-2c34-4e99-bd66-e98b304b36e7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:51.436095 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c97ca87-2c34-4e99-bd66-e98b304b36e7 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:51.436379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c97ca87-2c34-4e99-bd66-e98b304b36e7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:51.436651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c97ca87-2c34-4e99-bd66-e98b304b36e7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:51.438484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6c97ca87-2c34-4e99-bd66-e98b304b36e7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:51.507358 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c97ca87-2c34-4e99-bd66-e98b304b36e7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:51.508768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c97ca87-2c34-4e99-bd66-e98b304b36e7 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:51.518483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1275/2557] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:54.934977 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4163d9-6b3f-4458-91f8-e6b36b92c891 service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:54.936509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7b4163d9-6b3f-4458-91f8-e6b36b92c891 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:54.956256 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4163d9-6b3f-4458-91f8-e6b36b92c891 service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:54.963182 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f691d8bd-671c-4f67-8d58-a5b524c8e0a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:54.963817 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691d8bd-671c-4f67-8d58-a5b524c8e0a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:54.964338 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691d8bd-671c-4f67-8d58-a5b524c8e0a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:54.964338 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691d8bd-671c-4f67-8d58-a5b524c8e0a9 admin admin] PATH_INFO: `/v3/users/fbcd4c19e766446188e3b3f41a44e2e4` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:54.965558 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f691d8bd-671c-4f67-8d58-a5b524c8e0a9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fbcd4c19e766446188e3b3f41a44e2e4)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:54.967592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f691d8bd-671c-4f67-8d58-a5b524c8e0a9 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:55.154374 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f691d8bd-671c-4f67-8d58-a5b524c8e0a9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fbcd4c19e766446188e3b3f41a44e2e4'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:55.225249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f691d8bd-671c-4f67-8d58-a5b524c8e0a9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fbcd4c19e766446188e3b3f41a44e2e4'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:55.238274 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f691d8bd-671c-4f67-8d58-a5b524c8e0a9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fbcd4c19e766446188e3b3f41a44e2e4'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:55.238939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f691d8bd-671c-4f67-8d58-a5b524c8e0a9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fbcd4c19e766446188e3b3f41a44e2e4'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:55.246094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f691d8bd-671c-4f67-8d58-a5b524c8e0a9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fbcd4c19e766446188e3b3f41a44e2e4'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:55.252069 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1283/2558] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:54 2022] DELETE /identity/v3/users/fbcd4c19e766446188e3b3f41a44e2e4 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:55.513114 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c97ca87-2c34-4e99-bd66-e98b304b36e7 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:55.524723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c97ca87-2c34-4e99-bd66-e98b304b36e7 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:55.583163 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cc6937-f4b8-4431-a84a-14e6d26f0808 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:55.583797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cc6937-f4b8-4431-a84a-14e6d26f0808 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:55.584056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cc6937-f4b8-4431-a84a-14e6d26f0808 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:55.584324 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cc6937-f4b8-4431-a84a-14e6d26f0808 admin admin] PATH_INFO: `/v3/projects/55faed42b7f243df9df879d581797f56` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:55.585464 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cc6937-f4b8-4431-a84a-14e6d26f0808 admin admin] RBAC: Authorizing `identity:delete_project(project_id=55faed42b7f243df9df879d581797f56)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:55.587344 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cc6937-f4b8-4431-a84a-14e6d26f0808 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:55.683684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-11cc6937-f4b8-4431-a84a-14e6d26f0808 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '55faed42b7f243df9df879d581797f56', 'request_id': 'req-11cc6937-f4b8-4431-a84a-14e6d26f0808'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:55.689445 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-11cc6937-f4b8-4431-a84a-14e6d26f0808 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '55faed42b7f243df9df879d581797f56', 'request_id': 'req-11cc6937-f4b8-4431-a84a-14e6d26f0808'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:55.757107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-11cc6937-f4b8-4431-a84a-14e6d26f0808 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '55faed42b7f243df9df879d581797f56', 'request_id': 'req-11cc6937-f4b8-4431-a84a-14e6d26f0808'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:55.758500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:55.769818 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-11cc6937-f4b8-4431-a84a-14e6d26f0808 admin admin] The token cache is being invalidate because project 55faed42b7f243df9df879d581797f56 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:12:55.770179 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-11cc6937-f4b8-4431-a84a-14e6d26f0808 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:55.772874 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-11cc6937-f4b8-4431-a84a-14e6d26f0808 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:55.772874 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1276/2559] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:12:55 2022] DELETE /identity/v3/projects/55faed42b7f243df9df879d581797f56 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:55.777304 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba903ab-04c4-49ac-a71f-8966979f36f2 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:55.777594 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba903ab-04c4-49ac-a71f-8966979f36f2 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:55.777870 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba903ab-04c4-49ac-a71f-8966979f36f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:55.785484 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:55.798059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-aba903ab-04c4-49ac-a71f-8966979f36f2 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:55.799270 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-aba903ab-04c4-49ac-a71f-8966979f36f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:55.861419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1284/2560] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:55.870094 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63084584-86d1-4990-9d76-55840733c332 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:55.870425 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63084584-86d1-4990-9d76-55840733c332 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:55.870699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63084584-86d1-4990-9d76-55840733c332 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:55.890383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-63084584-86d1-4990-9d76-55840733c332 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:55.891580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-63084584-86d1-4990-9d76-55840733c332 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:55.899913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1277/2561] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:55.904299 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba903ab-04c4-49ac-a71f-8966979f36f2 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:55.914698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba903ab-04c4-49ac-a71f-8966979f36f2 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:55.919311 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7a7169-3637-4d06-88ae-f2fc14f64c4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:55.919905 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7a7169-3637-4d06-88ae-f2fc14f64c4f admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:55.920184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7a7169-3637-4d06-88ae-f2fc14f64c4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:55.920456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7a7169-3637-4d06-88ae-f2fc14f64c4f admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:55.921133 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7a7169-3637-4d06-88ae-f2fc14f64c4f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:12:55.921443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7a7169-3637-4d06-88ae-f2fc14f64c4f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:55.923043 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7a7169-3637-4d06-88ae-f2fc14f64c4f admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:55.934533 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1285/2562] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:12:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:12:55.938946 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63084584-86d1-4990-9d76-55840733c332 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:55.948972 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63084584-86d1-4990-9d76-55840733c332 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:55.954131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9e99ed-0e4e-4740-a961-56b47ffc7c2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:55.962325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9e99ed-0e4e-4740-a961-56b47ffc7c2f admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:55.962325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9e99ed-0e4e-4740-a961-56b47ffc7c2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:55.962325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9e99ed-0e4e-4740-a961-56b47ffc7c2f admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:55.962325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9e99ed-0e4e-4740-a961-56b47ffc7c2f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:55.962325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9e99ed-0e4e-4740-a961-56b47ffc7c2f admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:56.319332 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1278/2563] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:12:55 2022] POST /identity/v3/projects => generated 373 bytes in 381 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:56.326807 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7a7169-3637-4d06-88ae-f2fc14f64c4f admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:56.337706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7a7169-3637-4d06-88ae-f2fc14f64c4f admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:56.393411 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51f62ce-ebd8-4ee8-8930-786ff729d72a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:56.394077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51f62ce-ebd8-4ee8-8930-786ff729d72a admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:56.394359 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51f62ce-ebd8-4ee8-8930-786ff729d72a admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:56.394634 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51f62ce-ebd8-4ee8-8930-786ff729d72a admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:56.395297 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51f62ce-ebd8-4ee8-8930-786ff729d72a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:56.397594 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51f62ce-ebd8-4ee8-8930-786ff729d72a admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:56.545015 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1286/2564] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:12:56 2022] POST /identity/v3/users => generated 287 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:12:56.546127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9e99ed-0e4e-4740-a961-56b47ffc7c2f admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:56.558723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9e99ed-0e4e-4740-a961-56b47ffc7c2f admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:56.563634 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cb8900-ef03-4666-8857-26304dabb16e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:56.564353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cb8900-ef03-4666-8857-26304dabb16e admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:56.564719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cb8900-ef03-4666-8857-26304dabb16e admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:56.565064 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cb8900-ef03-4666-8857-26304dabb16e admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:56.566644 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88cb8900-ef03-4666-8857-26304dabb16e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:56.568699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88cb8900-ef03-4666-8857-26304dabb16e admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:56.582534 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1279/2565] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:56 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:56.584178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51f62ce-ebd8-4ee8-8930-786ff729d72a admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:56.595373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51f62ce-ebd8-4ee8-8930-786ff729d72a admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:56.611626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d73793b-ae29-4f9a-ba3e-6b0235c2c8e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:56.611626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d73793b-ae29-4f9a-ba3e-6b0235c2c8e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:56.611626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d73793b-ae29-4f9a-ba3e-6b0235c2c8e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:56.611626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d73793b-ae29-4f9a-ba3e-6b0235c2c8e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:56.612373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d73793b-ae29-4f9a-ba3e-6b0235c2c8e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:56.612373 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d73793b-ae29-4f9a-ba3e-6b0235c2c8e8 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:56.622607 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1287/2566] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:12:56 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:12:56.627030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cb8900-ef03-4666-8857-26304dabb16e admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:56.637974 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cb8900-ef03-4666-8857-26304dabb16e admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:56.643451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda45587-a94d-4133-956c-ac8df8429685 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:56.644275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda45587-a94d-4133-956c-ac8df8429685 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:56.644552 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda45587-a94d-4133-956c-ac8df8429685 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:56.644914 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda45587-a94d-4133-956c-ac8df8429685 admin admin] PATH_INFO: `/v3/projects/83428cdb97544c7b8ea4bf9a69d7106f/users/efa78ca640474aafbc99c03af1aca932/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:56.660241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda45587-a94d-4133-956c-ac8df8429685 admin admin] RBAC: Authorizing `identity:create_grant(project_id=83428cdb97544c7b8ea4bf9a69d7106f, user_id=efa78ca640474aafbc99c03af1aca932, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:56.663003 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda45587-a94d-4133-956c-ac8df8429685 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:56.758721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1280/2567] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:12:56 2022] PUT /identity/v3/projects/83428cdb97544c7b8ea4bf9a69d7106f/users/efa78ca640474aafbc99c03af1aca932/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:12:56.771249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70440d6-dd85-4496-9e26-d035d307785e None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:56.771531 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70440d6-dd85-4496-9e26-d035d307785e None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:56.771804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70440d6-dd85-4496-9e26-d035d307785e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:56.799302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-d70440d6-dd85-4496-9e26-d035d307785e None None] MFA Rules not processed for user `efa78ca640474aafbc99c03af1aca932`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:56.801338 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d70440d6-dd85-4496-9e26-d035d307785e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:56.862200 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1288/2568] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:56 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:56.874341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda45587-a94d-4133-956c-ac8df8429685 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:56.876442 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-fda45587-a94d-4133-956c-ac8df8429685 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:56.929189 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:56.949330 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda45587-a94d-4133-956c-ac8df8429685 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:56.954897 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c49879d-ce0f-44cf-9723-8d69825b3448 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:56.955540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c49879d-ce0f-44cf-9723-8d69825b3448 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:56.955821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c49879d-ce0f-44cf-9723-8d69825b3448 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:56.956093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c49879d-ce0f-44cf-9723-8d69825b3448 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:56.963911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c49879d-ce0f-44cf-9723-8d69825b3448 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:56.965488 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c49879d-ce0f-44cf-9723-8d69825b3448 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:56.974856 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1281/2569] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:56 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:57.893548 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70440d6-dd85-4496-9e26-d035d307785e None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:57.895168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d70440d6-dd85-4496-9e26-d035d307785e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:57.963818 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70440d6-dd85-4496-9e26-d035d307785e None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:57.968321 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4fabb9-aa66-40de-834d-1107e6bd004d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:57.968948 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4fabb9-aa66-40de-834d-1107e6bd004d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:57.969264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4fabb9-aa66-40de-834d-1107e6bd004d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:57.969540 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4fabb9-aa66-40de-834d-1107e6bd004d admin admin] PATH_INFO: `/v3/users/4bcf4aeb82bc4ad7b91a8fdb7f23bdfe` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:57.970663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4fabb9-aa66-40de-834d-1107e6bd004d admin admin] RBAC: Authorizing `identity:delete_user(user_id=4bcf4aeb82bc4ad7b91a8fdb7f23bdfe)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:57.972828 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4fabb9-aa66-40de-834d-1107e6bd004d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:58.155376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0c4fabb9-aa66-40de-834d-1107e6bd004d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4bcf4aeb82bc4ad7b91a8fdb7f23bdfe'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:58.271638 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0c4fabb9-aa66-40de-834d-1107e6bd004d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4bcf4aeb82bc4ad7b91a8fdb7f23bdfe'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:58.286259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0c4fabb9-aa66-40de-834d-1107e6bd004d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4bcf4aeb82bc4ad7b91a8fdb7f23bdfe'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:58.287047 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0c4fabb9-aa66-40de-834d-1107e6bd004d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4bcf4aeb82bc4ad7b91a8fdb7f23bdfe'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:58.294214 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-0c4fabb9-aa66-40de-834d-1107e6bd004d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4bcf4aeb82bc4ad7b91a8fdb7f23bdfe'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:58.300755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1289/2570] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:57 2022] DELETE /identity/v3/users/4bcf4aeb82bc4ad7b91a8fdb7f23bdfe => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:58.305374 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2815d742-378a-40b6-835d-f6d35148e01f None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:58.305867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2815d742-378a-40b6-835d-f6d35148e01f None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:58.305980 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2815d742-378a-40b6-835d-f6d35148e01f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:58.325349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-2815d742-378a-40b6-835d-f6d35148e01f None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:12:58.326578 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2815d742-378a-40b6-835d-f6d35148e01f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:58.399053 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1282/2571] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:12:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:12:58.408827 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4fabb9-aa66-40de-834d-1107e6bd004d admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:58.410814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0c4fabb9-aa66-40de-834d-1107e6bd004d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:58.471449 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4fabb9-aa66-40de-834d-1107e6bd004d admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:58.475855 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7390f14-c5f3-49c0-a452-a0c02b6b4bfa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:58.476619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7390f14-c5f3-49c0-a452-a0c02b6b4bfa service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:58.477013 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7390f14-c5f3-49c0-a452-a0c02b6b4bfa service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:58.478199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7390f14-c5f3-49c0-a452-a0c02b6b4bfa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:58.486675 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7390f14-c5f3-49c0-a452-a0c02b6b4bfa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:58.488736 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7390f14-c5f3-49c0-a452-a0c02b6b4bfa service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:58.498683 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1290/2572] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:12:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:12:59.166921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2815d742-378a-40b6-835d-f6d35148e01f None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:59.168739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2815d742-378a-40b6-835d-f6d35148e01f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:59.207457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2815d742-378a-40b6-835d-f6d35148e01f None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:59.207457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d548839-ec46-4697-bba4-5be8aa4aa2d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:59.207457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d548839-ec46-4697-bba4-5be8aa4aa2d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:59.207457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d548839-ec46-4697-bba4-5be8aa4aa2d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:59.208606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d548839-ec46-4697-bba4-5be8aa4aa2d8 admin admin] PATH_INFO: `/v3/projects/0a46e1b837a14345b3d11969f64f3dae` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:59.208606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d548839-ec46-4697-bba4-5be8aa4aa2d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a46e1b837a14345b3d11969f64f3dae)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:59.208606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d548839-ec46-4697-bba4-5be8aa4aa2d8 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:59.219113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:59.358297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0d548839-ec46-4697-bba4-5be8aa4aa2d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a46e1b837a14345b3d11969f64f3dae', 'request_id': 'req-0d548839-ec46-4697-bba4-5be8aa4aa2d8'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:59.364386 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0d548839-ec46-4697-bba4-5be8aa4aa2d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a46e1b837a14345b3d11969f64f3dae', 'request_id': 'req-0d548839-ec46-4697-bba4-5be8aa4aa2d8'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:59.460585 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0d548839-ec46-4697-bba4-5be8aa4aa2d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a46e1b837a14345b3d11969f64f3dae', 'request_id': 'req-0d548839-ec46-4697-bba4-5be8aa4aa2d8'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:59.472788 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0d548839-ec46-4697-bba4-5be8aa4aa2d8 admin admin] The token cache is being invalidate because project 0a46e1b837a14345b3d11969f64f3dae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:12:59.473154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0d548839-ec46-4697-bba4-5be8aa4aa2d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:59.473843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0d548839-ec46-4697-bba4-5be8aa4aa2d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:59.475327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1283/2573] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:12:59 2022] DELETE /identity/v3/projects/0a46e1b837a14345b3d11969f64f3dae => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:59.479397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7390f14-c5f3-49c0-a452-a0c02b6b4bfa service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:59.489797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7390f14-c5f3-49c0-a452-a0c02b6b4bfa service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:59.507301 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:12:59.546017 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263aad79-d95a-490b-ab65-94699bd73d92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:59.546638 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263aad79-d95a-490b-ab65-94699bd73d92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:59.546924 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263aad79-d95a-490b-ab65-94699bd73d92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:59.547200 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263aad79-d95a-490b-ab65-94699bd73d92 admin admin] PATH_INFO: `/v3/users/3a2e61adfbf74968a5a126e313ef461c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:59.548338 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263aad79-d95a-490b-ab65-94699bd73d92 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3a2e61adfbf74968a5a126e313ef461c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:59.550339 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263aad79-d95a-490b-ab65-94699bd73d92 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:59.687187 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d548839-ec46-4697-bba4-5be8aa4aa2d8 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:12:59.688675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-0d548839-ec46-4697-bba4-5be8aa4aa2d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:12:59.702373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d548839-ec46-4697-bba4-5be8aa4aa2d8 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:12:59.710898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76e92b0-9dec-40f3-af99-a6a08cf0ec63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:12:59.710898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e92b0-9dec-40f3-af99-a6a08cf0ec63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:12:59.710898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e92b0-9dec-40f3-af99-a6a08cf0ec63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:12:59.710898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76e92b0-9dec-40f3-af99-a6a08cf0ec63 admin admin] PATH_INFO: `/v3/users/a02db93f4e5d4255ac3c8f6d4729df35` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:12:59.711295 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76e92b0-9dec-40f3-af99-a6a08cf0ec63 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a02db93f4e5d4255ac3c8f6d4729df35)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:12:59.711295 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76e92b0-9dec-40f3-af99-a6a08cf0ec63 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:12:59.765401 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-263aad79-d95a-490b-ab65-94699bd73d92 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3a2e61adfbf74968a5a126e313ef461c'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:59.877155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-263aad79-d95a-490b-ab65-94699bd73d92 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3a2e61adfbf74968a5a126e313ef461c'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:59.890133 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-263aad79-d95a-490b-ab65-94699bd73d92 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3a2e61adfbf74968a5a126e313ef461c'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:59.890779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-263aad79-d95a-490b-ab65-94699bd73d92 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3a2e61adfbf74968a5a126e313ef461c'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:59.897743 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-263aad79-d95a-490b-ab65-94699bd73d92 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3a2e61adfbf74968a5a126e313ef461c'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:59.903776 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1291/2574] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:59 2022] DELETE /identity/v3/users/3a2e61adfbf74968a5a126e313ef461c => generated 0 bytes in 425 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:12:59.937778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e76e92b0-9dec-40f3-af99-a6a08cf0ec63 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a02db93f4e5d4255ac3c8f6d4729df35'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:59.938538 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e76e92b0-9dec-40f3-af99-a6a08cf0ec63 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a02db93f4e5d4255ac3c8f6d4729df35'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:12:59.997372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e76e92b0-9dec-40f3-af99-a6a08cf0ec63 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a02db93f4e5d4255ac3c8f6d4729df35'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.010034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e76e92b0-9dec-40f3-af99-a6a08cf0ec63 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a02db93f4e5d4255ac3c8f6d4729df35'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.010691 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e76e92b0-9dec-40f3-af99-a6a08cf0ec63 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a02db93f4e5d4255ac3c8f6d4729df35'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.023088 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1284/2575] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:12:59 2022] DELETE /identity/v3/users/a02db93f4e5d4255ac3c8f6d4729df35 => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:13:00.027530 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaeb2317-6539-42c9-bdea-656bed82d2ee None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:00.027819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaeb2317-6539-42c9-bdea-656bed82d2ee None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:00.028096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaeb2317-6539-42c9-bdea-656bed82d2ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:00.047818 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-eaeb2317-6539-42c9-bdea-656bed82d2ee None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:00.049030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-eaeb2317-6539-42c9-bdea-656bed82d2ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:00.110068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1292/2576] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:00.133145 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76e92b0-9dec-40f3-af99-a6a08cf0ec63 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:00.134599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e76e92b0-9dec-40f3-af99-a6a08cf0ec63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:00.194850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76e92b0-9dec-40f3-af99-a6a08cf0ec63 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:00.198539 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae7bb67-94c8-4c28-a3fa-f5ddf4af5647 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:00.199147 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae7bb67-94c8-4c28-a3fa-f5ddf4af5647 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:00.199423 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae7bb67-94c8-4c28-a3fa-f5ddf4af5647 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:00.199693 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae7bb67-94c8-4c28-a3fa-f5ddf4af5647 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:00.210305 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae7bb67-94c8-4c28-a3fa-f5ddf4af5647 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:00.216510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae7bb67-94c8-4c28-a3fa-f5ddf4af5647 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:00.225937 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1285/2577] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:13:00.259882 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaeb2317-6539-42c9-bdea-656bed82d2ee None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:00.261338 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-eaeb2317-6539-42c9-bdea-656bed82d2ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:00.273510 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaeb2317-6539-42c9-bdea-656bed82d2ee None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:00.282010 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfcca6d-8ce4-4e94-b6f7-e681d63d8afc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:00.282010 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfcca6d-8ce4-4e94-b6f7-e681d63d8afc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:00.282010 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfcca6d-8ce4-4e94-b6f7-e681d63d8afc admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:00.282010 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfcca6d-8ce4-4e94-b6f7-e681d63d8afc admin admin] PATH_INFO: `/v3/users/efa78ca640474aafbc99c03af1aca932` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:00.282374 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfcca6d-8ce4-4e94-b6f7-e681d63d8afc admin admin] RBAC: Authorizing `identity:delete_user(user_id=efa78ca640474aafbc99c03af1aca932)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:00.282374 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfcca6d-8ce4-4e94-b6f7-e681d63d8afc admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:00.343472 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae7bb67-94c8-4c28-a3fa-f5ddf4af5647 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:00.344966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8ae7bb67-94c8-4c28-a3fa-f5ddf4af5647 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:00.358685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae7bb67-94c8-4c28-a3fa-f5ddf4af5647 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:00.364546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1012ad-f7a6-40fd-ae3e-350b14a44eed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:00.365237 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1012ad-f7a6-40fd-ae3e-350b14a44eed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:00.368590 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1012ad-f7a6-40fd-ae3e-350b14a44eed admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:00.368590 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1012ad-f7a6-40fd-ae3e-350b14a44eed admin admin] PATH_INFO: `/v3/projects/fdb2b7eabf214f66a355bd53eebdb9f5` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:00.368590 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1012ad-f7a6-40fd-ae3e-350b14a44eed admin admin] RBAC: Authorizing `identity:delete_project(project_id=fdb2b7eabf214f66a355bd53eebdb9f5)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:00.369269 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1012ad-f7a6-40fd-ae3e-350b14a44eed admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:00.386550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:00.436552 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:00.445844 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7dfcca6d-8ce4-4e94-b6f7-e681d63d8afc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'efa78ca640474aafbc99c03af1aca932'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.480635 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-db1012ad-f7a6-40fd-ae3e-350b14a44eed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fdb2b7eabf214f66a355bd53eebdb9f5', 'request_id': 'req-db1012ad-f7a6-40fd-ae3e-350b14a44eed'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.486249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-db1012ad-f7a6-40fd-ae3e-350b14a44eed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fdb2b7eabf214f66a355bd53eebdb9f5', 'request_id': 'req-db1012ad-f7a6-40fd-ae3e-350b14a44eed'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.501572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7dfcca6d-8ce4-4e94-b6f7-e681d63d8afc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'efa78ca640474aafbc99c03af1aca932'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.515936 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7dfcca6d-8ce4-4e94-b6f7-e681d63d8afc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'efa78ca640474aafbc99c03af1aca932'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.516803 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7dfcca6d-8ce4-4e94-b6f7-e681d63d8afc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'efa78ca640474aafbc99c03af1aca932'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.526175 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7dfcca6d-8ce4-4e94-b6f7-e681d63d8afc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'efa78ca640474aafbc99c03af1aca932'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.532187 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1293/2578] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:13:00 2022] DELETE /identity/v3/users/efa78ca640474aafbc99c03af1aca932 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:13:00.536601 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920addfa-7f87-486a-b508-c268e2a86f2b None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:00.541667 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920addfa-7f87-486a-b508-c268e2a86f2b None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:00.541667 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920addfa-7f87-486a-b508-c268e2a86f2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:00.556744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-920addfa-7f87-486a-b508-c268e2a86f2b None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:00.558008 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-920addfa-7f87-486a-b508-c268e2a86f2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:00.563353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-db1012ad-f7a6-40fd-ae3e-350b14a44eed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fdb2b7eabf214f66a355bd53eebdb9f5', 'request_id': 'req-db1012ad-f7a6-40fd-ae3e-350b14a44eed'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.576865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-db1012ad-f7a6-40fd-ae3e-350b14a44eed admin admin] The token cache is being invalidate because project fdb2b7eabf214f66a355bd53eebdb9f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:13:00.577463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-db1012ad-f7a6-40fd-ae3e-350b14a44eed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.578107 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-db1012ad-f7a6-40fd-ae3e-350b14a44eed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.580486 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1286/2579] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:13:00 2022] DELETE /identity/v3/projects/fdb2b7eabf214f66a355bd53eebdb9f5 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:13:00.585968 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499212d8-0baf-402e-aac3-2cd194fc5124 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:00.586255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499212d8-0baf-402e-aac3-2cd194fc5124 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:00.586536 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499212d8-0baf-402e-aac3-2cd194fc5124 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:00.608960 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-499212d8-0baf-402e-aac3-2cd194fc5124 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:00.610248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-499212d8-0baf-402e-aac3-2cd194fc5124 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:00.625514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1294/2580] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:13:00.637193 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920addfa-7f87-486a-b508-c268e2a86f2b None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:00.638642 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-920addfa-7f87-486a-b508-c268e2a86f2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:00.682802 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1287/2581] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:00.690773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b678b76-636a-4ace-ac9a-f2e66f3f8309 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:00.691067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b678b76-636a-4ace-ac9a-f2e66f3f8309 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:00.691881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b678b76-636a-4ace-ac9a-f2e66f3f8309 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:00.706115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920addfa-7f87-486a-b508-c268e2a86f2b None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:00.711738 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2636b0a9-13ef-4215-962a-cde50b6a4e16 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:00.712361 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2636b0a9-13ef-4215-962a-cde50b6a4e16 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:00.712560 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-8b678b76-636a-4ace-ac9a-f2e66f3f8309 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:00.712686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2636b0a9-13ef-4215-962a-cde50b6a4e16 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:00.712959 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2636b0a9-13ef-4215-962a-cde50b6a4e16 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:00.714010 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8b678b76-636a-4ace-ac9a-f2e66f3f8309 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:00.722320 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1288/2582] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:13:00.723014 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b678b76-636a-4ace-ac9a-f2e66f3f8309 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:00.724325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8b678b76-636a-4ace-ac9a-f2e66f3f8309 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:00.725094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2636b0a9-13ef-4215-962a-cde50b6a4e16 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:00.726899 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2636b0a9-13ef-4215-962a-cde50b6a4e16 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:00.736935 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b678b76-636a-4ace-ac9a-f2e66f3f8309 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:00.737416 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1295/2583] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:13:00.738135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2636b0a9-13ef-4215-962a-cde50b6a4e16 service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:00.749556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4841e72f-4541-4601-a908-f58c358fb6d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:00.749556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4841e72f-4541-4601-a908-f58c358fb6d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:00.749556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4841e72f-4541-4601-a908-f58c358fb6d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:00.749556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4841e72f-4541-4601-a908-f58c358fb6d2 admin admin] PATH_INFO: `/v3/projects/0af4373504224763a31adfd64271d0a2` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:00.750127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4841e72f-4541-4601-a908-f58c358fb6d2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0af4373504224763a31adfd64271d0a2)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:00.752195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4841e72f-4541-4601-a908-f58c358fb6d2 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:00.755778 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2636b0a9-13ef-4215-962a-cde50b6a4e16 service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:00.761503 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c00d63-9053-427f-b608-97e68265f5c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:00.762454 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c00d63-9053-427f-b608-97e68265f5c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:00.769199 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c00d63-9053-427f-b608-97e68265f5c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:00.769636 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c00d63-9053-427f-b608-97e68265f5c5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:00.770932 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c00d63-9053-427f-b608-97e68265f5c5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:13:00.771279 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c00d63-9053-427f-b608-97e68265f5c5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:00.773178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c00d63-9053-427f-b608-97e68265f5c5 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:00.785927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1296/2584] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:13:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:13:00.790207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c00d63-9053-427f-b608-97e68265f5c5 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:00.800481 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c00d63-9053-427f-b608-97e68265f5c5 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:00.805285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b1c474-f5c8-4030-b799-3aba54ba1c11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:00.805885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b1c474-f5c8-4030-b799-3aba54ba1c11 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:00.806165 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b1c474-f5c8-4030-b799-3aba54ba1c11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:00.806441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b1c474-f5c8-4030-b799-3aba54ba1c11 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:00.807206 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b1c474-f5c8-4030-b799-3aba54ba1c11 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:00.809745 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b1c474-f5c8-4030-b799-3aba54ba1c11 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:00.884501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-4841e72f-4541-4601-a908-f58c358fb6d2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0af4373504224763a31adfd64271d0a2', 'request_id': 'req-4841e72f-4541-4601-a908-f58c358fb6d2'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.890339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-4841e72f-4541-4601-a908-f58c358fb6d2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0af4373504224763a31adfd64271d0a2', 'request_id': 'req-4841e72f-4541-4601-a908-f58c358fb6d2'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.895396 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1297/2585] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:13:00 2022] POST /identity/v3/projects => generated 387 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:13:00.899563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b1c474-f5c8-4030-b799-3aba54ba1c11 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:00.920882 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:00.957532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-4841e72f-4541-4601-a908-f58c358fb6d2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0af4373504224763a31adfd64271d0a2', 'request_id': 'req-4841e72f-4541-4601-a908-f58c358fb6d2'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.964693 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b1c474-f5c8-4030-b799-3aba54ba1c11 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:00.969875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983feea0-619b-4199-b3c0-cb64154cd537 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:00.970494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983feea0-619b-4199-b3c0-cb64154cd537 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:00.970777 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983feea0-619b-4199-b3c0-cb64154cd537 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:00.971189 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983feea0-619b-4199-b3c0-cb64154cd537 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:00.971333 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-4841e72f-4541-4601-a908-f58c358fb6d2 admin admin] The token cache is being invalidate because project 0af4373504224763a31adfd64271d0a2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:13:00.971712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-4841e72f-4541-4601-a908-f58c358fb6d2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.971841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983feea0-619b-4199-b3c0-cb64154cd537 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:00.972615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-4841e72f-4541-4601-a908-f58c358fb6d2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:00.974326 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983feea0-619b-4199-b3c0-cb64154cd537 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:00.975061 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1289/2586] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:13:00 2022] DELETE /identity/v3/projects/0af4373504224763a31adfd64271d0a2 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:00.980523 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4841e72f-4541-4601-a908-f58c358fb6d2 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:00.982672 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-4841e72f-4541-4601-a908-f58c358fb6d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:01.001952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:01.053638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4841e72f-4541-4601-a908-f58c358fb6d2 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:01.059511 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4494ce92-93ef-49d7-bf57-e8017c05e319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:01.060148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4494ce92-93ef-49d7-bf57-e8017c05e319 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:01.060473 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4494ce92-93ef-49d7-bf57-e8017c05e319 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:01.060690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4494ce92-93ef-49d7-bf57-e8017c05e319 admin admin] PATH_INFO: `/v3/users/1cd350479e314e069257c2c19e019a98` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:01.061870 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4494ce92-93ef-49d7-bf57-e8017c05e319 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1cd350479e314e069257c2c19e019a98)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:01.063778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4494ce92-93ef-49d7-bf57-e8017c05e319 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:01.067787 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1298/2587] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:13:00 2022] POST /identity/v3/users => generated 294 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:13:01.071825 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983feea0-619b-4199-b3c0-cb64154cd537 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:01.073886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-983feea0-619b-4199-b3c0-cb64154cd537 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:01.086644 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983feea0-619b-4199-b3c0-cb64154cd537 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:01.092719 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76fbbee-b1be-4bcc-8f1d-9f0f0ffb8e97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:01.093404 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76fbbee-b1be-4bcc-8f1d-9f0f0ffb8e97 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:01.093691 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76fbbee-b1be-4bcc-8f1d-9f0f0ffb8e97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:01.094001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76fbbee-b1be-4bcc-8f1d-9f0f0ffb8e97 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:01.095227 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76fbbee-b1be-4bcc-8f1d-9f0f0ffb8e97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:01.096292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76fbbee-b1be-4bcc-8f1d-9f0f0ffb8e97 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:01.107605 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1299/2588] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:01 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:13:01.111449 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76fbbee-b1be-4bcc-8f1d-9f0f0ffb8e97 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:01.122252 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76fbbee-b1be-4bcc-8f1d-9f0f0ffb8e97 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:01.126914 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913bcd59-bd62-4bf0-9057-75eaa7a43e3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:01.127470 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913bcd59-bd62-4bf0-9057-75eaa7a43e3d admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:01.127889 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913bcd59-bd62-4bf0-9057-75eaa7a43e3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:01.128170 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913bcd59-bd62-4bf0-9057-75eaa7a43e3d admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:01.128814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913bcd59-bd62-4bf0-9057-75eaa7a43e3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:01.131126 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913bcd59-bd62-4bf0-9057-75eaa7a43e3d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:01.141822 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1300/2589] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:01 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:13:01.145915 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913bcd59-bd62-4bf0-9057-75eaa7a43e3d admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:01.155500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913bcd59-bd62-4bf0-9057-75eaa7a43e3d admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:01.160135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdb53cf-525e-4d86-b9fc-7bc5c32721a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:01.160941 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdb53cf-525e-4d86-b9fc-7bc5c32721a2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:01.161432 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdb53cf-525e-4d86-b9fc-7bc5c32721a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:01.161714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdb53cf-525e-4d86-b9fc-7bc5c32721a2 admin admin] PATH_INFO: `/v3/projects/c42193bc84da4389a3c050979cd759e3/users/f9e3043364ef49ca94e41297bd02987f/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:01.176774 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdb53cf-525e-4d86-b9fc-7bc5c32721a2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c42193bc84da4389a3c050979cd759e3, user_id=f9e3043364ef49ca94e41297bd02987f, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:01.179482 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdb53cf-525e-4d86-b9fc-7bc5c32721a2 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:01.250064 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1301/2590] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:13:01 2022] PUT /identity/v3/projects/c42193bc84da4389a3c050979cd759e3/users/f9e3043364ef49ca94e41297bd02987f/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:13:01.250972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdb53cf-525e-4d86-b9fc-7bc5c32721a2 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:01.252710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-afdb53cf-525e-4d86-b9fc-7bc5c32721a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:01.302567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-4494ce92-93ef-49d7-bf57-e8017c05e319 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1cd350479e314e069257c2c19e019a98'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:01.303386 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-4494ce92-93ef-49d7-bf57-e8017c05e319 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1cd350479e314e069257c2c19e019a98'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:01.319314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdb53cf-525e-4d86-b9fc-7bc5c32721a2 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:01.323897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349a1bac-08d7-4dcb-aea2-fec83e6c72b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:01.324543 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349a1bac-08d7-4dcb-aea2-fec83e6c72b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:01.324790 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349a1bac-08d7-4dcb-aea2-fec83e6c72b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:01.325055 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349a1bac-08d7-4dcb-aea2-fec83e6c72b6 admin admin] PATH_INFO: `/v3/projects/83428cdb97544c7b8ea4bf9a69d7106f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:01.326230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349a1bac-08d7-4dcb-aea2-fec83e6c72b6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=83428cdb97544c7b8ea4bf9a69d7106f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:01.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349a1bac-08d7-4dcb-aea2-fec83e6c72b6 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:01.356831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-4494ce92-93ef-49d7-bf57-e8017c05e319 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1cd350479e314e069257c2c19e019a98'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:01.369733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-4494ce92-93ef-49d7-bf57-e8017c05e319 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1cd350479e314e069257c2c19e019a98'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:01.370394 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-4494ce92-93ef-49d7-bf57-e8017c05e319 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1cd350479e314e069257c2c19e019a98'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:01.382999 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1290/2591] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:13:00 2022] DELETE /identity/v3/users/1cd350479e314e069257c2c19e019a98 => generated 0 bytes in 404 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:13:01.387207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53386a1-5e7a-4add-b2a7-48cc156ff9a2 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:01.387501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53386a1-5e7a-4add-b2a7-48cc156ff9a2 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:01.387777 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53386a1-5e7a-4add-b2a7-48cc156ff9a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:01.417555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-e53386a1-5e7a-4add-b2a7-48cc156ff9a2 None None] MFA Rules not processed for user `f9e3043364ef49ca94e41297bd02987f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:01.418021 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-349a1bac-08d7-4dcb-aea2-fec83e6c72b6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '83428cdb97544c7b8ea4bf9a69d7106f', 'request_id': 'req-349a1bac-08d7-4dcb-aea2-fec83e6c72b6'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:01.419276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e53386a1-5e7a-4add-b2a7-48cc156ff9a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:01.430636 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-349a1bac-08d7-4dcb-aea2-fec83e6c72b6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '83428cdb97544c7b8ea4bf9a69d7106f', 'request_id': 'req-349a1bac-08d7-4dcb-aea2-fec83e6c72b6'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:01.438892 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-349a1bac-08d7-4dcb-aea2-fec83e6c72b6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '83428cdb97544c7b8ea4bf9a69d7106f', 'request_id': 'req-349a1bac-08d7-4dcb-aea2-fec83e6c72b6'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:01.442482 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:INSERT=5,keystone:DELETE=6 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:01.492294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1291/2592] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:01 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:01.496600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53386a1-5e7a-4add-b2a7-48cc156ff9a2 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:01.497965 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e53386a1-5e7a-4add-b2a7-48cc156ff9a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:01.526387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-349a1bac-08d7-4dcb-aea2-fec83e6c72b6 admin admin] The token cache is being invalidate because project 83428cdb97544c7b8ea4bf9a69d7106f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:13:01.526387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-349a1bac-08d7-4dcb-aea2-fec83e6c72b6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:01.526387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-349a1bac-08d7-4dcb-aea2-fec83e6c72b6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:01.526387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1302/2593] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:13:01 2022] DELETE /identity/v3/projects/83428cdb97544c7b8ea4bf9a69d7106f => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:01.527146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:01.530188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc16b32-3707-41f8-b6c8-81d5033c0edd None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:01.530477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc16b32-3707-41f8-b6c8-81d5033c0edd None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:01.530755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc16b32-3707-41f8-b6c8-81d5033c0edd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:01.569454 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53386a1-5e7a-4add-b2a7-48cc156ff9a2 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:01.576349 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06912d67-4c91-4fdf-a60f-6bcfb945880c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:01.577301 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06912d67-4c91-4fdf-a60f-6bcfb945880c admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:01.577591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06912d67-4c91-4fdf-a60f-6bcfb945880c admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:01.577866 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06912d67-4c91-4fdf-a60f-6bcfb945880c admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:01.578500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06912d67-4c91-4fdf-a60f-6bcfb945880c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:01.580436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06912d67-4c91-4fdf-a60f-6bcfb945880c admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:01.595283 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-efc16b32-3707-41f8-b6c8-81d5033c0edd None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:01.596697 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-efc16b32-3707-41f8-b6c8-81d5033c0edd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:01.607455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1303/2594] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:13:01.621334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246b40e0-d514-43c1-9eb4-708670907f3b None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:01.621628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246b40e0-d514-43c1-9eb4-708670907f3b None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:01.621908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246b40e0-d514-43c1-9eb4-708670907f3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:01.645232 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-246b40e0-d514-43c1-9eb4-708670907f3b None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:01.646560 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-246b40e0-d514-43c1-9eb4-708670907f3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:01.655593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1304/2595] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:13:01.659514 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246b40e0-d514-43c1-9eb4-708670907f3b None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:01.668592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246b40e0-d514-43c1-9eb4-708670907f3b None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:01.673570 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3ede3d-1b8a-484f-bb65-e01db7fb28a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:01.674171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3ede3d-1b8a-484f-bb65-e01db7fb28a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:01.674444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3ede3d-1b8a-484f-bb65-e01db7fb28a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:01.674711 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3ede3d-1b8a-484f-bb65-e01db7fb28a6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:01.675414 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3ede3d-1b8a-484f-bb65-e01db7fb28a6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:13:01.676366 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3ede3d-1b8a-484f-bb65-e01db7fb28a6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:01.678009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3ede3d-1b8a-484f-bb65-e01db7fb28a6 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:01.691372 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1305/2596] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:13:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 01:13:01.698154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1292/2597] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:13:01 2022] POST /identity/v3/projects => generated 385 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:13:01.699403 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3ede3d-1b8a-484f-bb65-e01db7fb28a6 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:01.699747 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06912d67-4c91-4fdf-a60f-6bcfb945880c admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:01.703091 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-06912d67-4c91-4fdf-a60f-6bcfb945880c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:01.715438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3ede3d-1b8a-484f-bb65-e01db7fb28a6 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:01.771927 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06912d67-4c91-4fdf-a60f-6bcfb945880c admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:01.772873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468a329f-bea3-47bf-be84-114a49d5df8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:01.773582 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468a329f-bea3-47bf-be84-114a49d5df8a admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:01.773864 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468a329f-bea3-47bf-be84-114a49d5df8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:01.774133 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468a329f-bea3-47bf-be84-114a49d5df8a admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:01.774760 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468a329f-bea3-47bf-be84-114a49d5df8a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:01.776724 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468a329f-bea3-47bf-be84-114a49d5df8a admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:01.777543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7d62d9-b33c-401c-bdaf-bc2050db2f77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:01.778143 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7d62d9-b33c-401c-bdaf-bc2050db2f77 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:01.778440 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7d62d9-b33c-401c-bdaf-bc2050db2f77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:01.778699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7d62d9-b33c-401c-bdaf-bc2050db2f77 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:01.779328 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7d62d9-b33c-401c-bdaf-bc2050db2f77 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:01.781292 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7d62d9-b33c-401c-bdaf-bc2050db2f77 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:01.900367 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1293/2598] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:13:01 2022] POST /identity/v3/users => generated 299 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:01.901244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7d62d9-b33c-401c-bdaf-bc2050db2f77 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:01.903626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1306/2599] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:13:01 2022] POST /identity/v3/projects => generated 369 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:13:01.903784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-aa7d62d9-b33c-401c-bdaf-bc2050db2f77 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:01.904801 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468a329f-bea3-47bf-be84-114a49d5df8a admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:01.915912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7d62d9-b33c-401c-bdaf-bc2050db2f77 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:01.968234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468a329f-bea3-47bf-be84-114a49d5df8a admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:01.972035 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155d4444-ec38-4849-b77b-876500e9c9c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:01.973249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155d4444-ec38-4849-b77b-876500e9c9c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:01.973716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155d4444-ec38-4849-b77b-876500e9c9c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:01.973998 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155d4444-ec38-4849-b77b-876500e9c9c0 admin admin] PATH_INFO: `/v3/projects/15ed0cea1cd345429921b1083d321708` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:01.975159 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155d4444-ec38-4849-b77b-876500e9c9c0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=15ed0cea1cd345429921b1083d321708)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:01.977160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155d4444-ec38-4849-b77b-876500e9c9c0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:01.977965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf7fa59-b69b-4b3c-97b3-009d775c28e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:01.978629 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf7fa59-b69b-4b3c-97b3-009d775c28e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:01.978983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf7fa59-b69b-4b3c-97b3-009d775c28e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:01.980796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf7fa59-b69b-4b3c-97b3-009d775c28e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:01.981496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf7fa59-b69b-4b3c-97b3-009d775c28e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:01.983108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf7fa59-b69b-4b3c-97b3-009d775c28e7 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:01.995780 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1307/2600] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:01 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:13:01.997543 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf7fa59-b69b-4b3c-97b3-009d775c28e7 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:02.008783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf7fa59-b69b-4b3c-97b3-009d775c28e7 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:02.013517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e64687-8ebf-4610-8494-c465086d8bee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:02.014106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e64687-8ebf-4610-8494-c465086d8bee admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:02.014417 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e64687-8ebf-4610-8494-c465086d8bee admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:02.014704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e64687-8ebf-4610-8494-c465086d8bee admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:02.015355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e64687-8ebf-4610-8494-c465086d8bee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:02.017286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e64687-8ebf-4610-8494-c465086d8bee admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:02.071162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:02.078511 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-155d4444-ec38-4849-b77b-876500e9c9c0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '15ed0cea1cd345429921b1083d321708', 'request_id': 'req-155d4444-ec38-4849-b77b-876500e9c9c0'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:02.084223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:DELETE=1,keystone:SELECT=95,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:02.088087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-155d4444-ec38-4849-b77b-876500e9c9c0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '15ed0cea1cd345429921b1083d321708', 'request_id': 'req-155d4444-ec38-4849-b77b-876500e9c9c0'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:02.089598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1308/2601] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:13:01 2022] POST /identity/v3/users => generated 285 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:02.090377 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e64687-8ebf-4610-8494-c465086d8bee admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:02.100672 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e64687-8ebf-4610-8494-c465086d8bee admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:02.151522 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-155d4444-ec38-4849-b77b-876500e9c9c0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '15ed0cea1cd345429921b1083d321708', 'request_id': 'req-155d4444-ec38-4849-b77b-876500e9c9c0'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:02.155656 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203d3ba4-48d2-4198-b994-686bf8dcb0e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:02.156332 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203d3ba4-48d2-4198-b994-686bf8dcb0e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:02.156657 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203d3ba4-48d2-4198-b994-686bf8dcb0e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:02.156993 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203d3ba4-48d2-4198-b994-686bf8dcb0e3 admin admin] PATH_INFO: `/v3/projects/e66308011618464f9e6519da298e7d71/users/6113a6d40c0047e1b1ed2849ac6349b4/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:02.163733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-155d4444-ec38-4849-b77b-876500e9c9c0 admin admin] The token cache is being invalidate because project 15ed0cea1cd345429921b1083d321708 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:13:02.164025 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-155d4444-ec38-4849-b77b-876500e9c9c0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:02.164697 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-155d4444-ec38-4849-b77b-876500e9c9c0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:02.166233 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1294/2602] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:13:01 2022] DELETE /identity/v3/projects/15ed0cea1cd345429921b1083d321708 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:02.167003 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155d4444-ec38-4849-b77b-876500e9c9c0 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:02.168436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-155d4444-ec38-4849-b77b-876500e9c9c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:02.174150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203d3ba4-48d2-4198-b994-686bf8dcb0e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e66308011618464f9e6519da298e7d71, user_id=6113a6d40c0047e1b1ed2849ac6349b4, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:02.177610 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203d3ba4-48d2-4198-b994-686bf8dcb0e3 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:02.181450 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155d4444-ec38-4849-b77b-876500e9c9c0 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:02.186171 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3824a73c-e4ee-4cc5-b380-c332dd88aebe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:02.186781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3824a73c-e4ee-4cc5-b380-c332dd88aebe admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:02.187049 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3824a73c-e4ee-4cc5-b380-c332dd88aebe admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:02.187323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3824a73c-e4ee-4cc5-b380-c332dd88aebe admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:02.187969 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3824a73c-e4ee-4cc5-b380-c332dd88aebe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:02.189607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3824a73c-e4ee-4cc5-b380-c332dd88aebe admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:02.199610 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1295/2603] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:02 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:02.200747 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9367121f-6469-4ee4-a0b3-b028235e30dd None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:02.201053 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9367121f-6469-4ee4-a0b3-b028235e30dd None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:02.206462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9367121f-6469-4ee4-a0b3-b028235e30dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:02.220970 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-9367121f-6469-4ee4-a0b3-b028235e30dd None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:02.222204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-9367121f-6469-4ee4-a0b3-b028235e30dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:02.230747 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1296/2604] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:02.231517 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9367121f-6469-4ee4-a0b3-b028235e30dd None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:02.241491 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9367121f-6469-4ee4-a0b3-b028235e30dd None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:02.246302 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05261db8-b7b1-4d40-a622-6d081887d5ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:02.246991 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05261db8-b7b1-4d40-a622-6d081887d5ae admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:02.247214 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05261db8-b7b1-4d40-a622-6d081887d5ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:02.247712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05261db8-b7b1-4d40-a622-6d081887d5ae admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:02.248365 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05261db8-b7b1-4d40-a622-6d081887d5ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:02.250253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05261db8-b7b1-4d40-a622-6d081887d5ae admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:02.260462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1297/2605] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:02 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:02.261605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d0974a-de03-43f0-9871-7462de0c9a3b None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:02.261887 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d0974a-de03-43f0-9871-7462de0c9a3b None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:02.262161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d0974a-de03-43f0-9871-7462de0c9a3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:02.280767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-62d0974a-de03-43f0-9871-7462de0c9a3b None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:02.282076 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-62d0974a-de03-43f0-9871-7462de0c9a3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:02.290729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1298/2606] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:02.291471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d0974a-de03-43f0-9871-7462de0c9a3b None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:02.301636 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1309/2607] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:13:02 2022] PUT /identity/v3/projects/e66308011618464f9e6519da298e7d71/users/6113a6d40c0047e1b1ed2849ac6349b4/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:02.301949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d0974a-de03-43f0-9871-7462de0c9a3b None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:02.302639 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203d3ba4-48d2-4198-b994-686bf8dcb0e3 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:02.363327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75da027f-3398-4556-b1da-bf8c1738bbf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:02.364072 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75da027f-3398-4556-b1da-bf8c1738bbf8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:02.364356 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75da027f-3398-4556-b1da-bf8c1738bbf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:02.364630 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75da027f-3398-4556-b1da-bf8c1738bbf8 admin admin] PATH_INFO: `/v3/projects/66afded63f464752aa18426be3a437ac/users/2c1fe639fb5847fe9bbe1303b8688215/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:02.373190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203d3ba4-48d2-4198-b994-686bf8dcb0e3 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:02.378067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955aa0c7-5154-4d75-b3ae-173ddac69df5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:02.378673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955aa0c7-5154-4d75-b3ae-173ddac69df5 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:02.378995 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955aa0c7-5154-4d75-b3ae-173ddac69df5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:02.379424 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955aa0c7-5154-4d75-b3ae-173ddac69df5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:02.380358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955aa0c7-5154-4d75-b3ae-173ddac69df5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:13:02.380549 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75da027f-3398-4556-b1da-bf8c1738bbf8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=66afded63f464752aa18426be3a437ac, user_id=2c1fe639fb5847fe9bbe1303b8688215, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:02.380997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955aa0c7-5154-4d75-b3ae-173ddac69df5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:02.382407 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955aa0c7-5154-4d75-b3ae-173ddac69df5 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:02.384719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75da027f-3398-4556-b1da-bf8c1738bbf8 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:02.393929 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1310/2608] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:13:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:13:02.395683 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955aa0c7-5154-4d75-b3ae-173ddac69df5 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:02.397447 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-955aa0c7-5154-4d75-b3ae-173ddac69df5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:02.411771 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955aa0c7-5154-4d75-b3ae-173ddac69df5 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:02.416413 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8868bea6-ed24-44c8-a1c2-b6f97bedad13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:02.416996 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8868bea6-ed24-44c8-a1c2-b6f97bedad13 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:02.417312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8868bea6-ed24-44c8-a1c2-b6f97bedad13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:02.417591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8868bea6-ed24-44c8-a1c2-b6f97bedad13 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:02.418227 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8868bea6-ed24-44c8-a1c2-b6f97bedad13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:02.419802 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8868bea6-ed24-44c8-a1c2-b6f97bedad13 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:02.430354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1311/2609] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:02 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:02.431076 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8868bea6-ed24-44c8-a1c2-b6f97bedad13 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:02.441174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8868bea6-ed24-44c8-a1c2-b6f97bedad13 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:02.443921 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1299/2610] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:13:02 2022] PUT /identity/v3/projects/66afded63f464752aa18426be3a437ac/users/2c1fe639fb5847fe9bbe1303b8688215/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:02.448053 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75da027f-3398-4556-b1da-bf8c1738bbf8 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:02.509246 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8fb77c-9aa4-4327-b750-2caced7f705a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:02.511104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8fb77c-9aa4-4327-b750-2caced7f705a admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:02.511104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8fb77c-9aa4-4327-b750-2caced7f705a admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:02.511104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8fb77c-9aa4-4327-b750-2caced7f705a admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:02.511402 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75da027f-3398-4556-b1da-bf8c1738bbf8 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:02.511500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8fb77c-9aa4-4327-b750-2caced7f705a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:02.512923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8fb77c-9aa4-4327-b750-2caced7f705a admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:02.515901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2ae3a6-fe80-4bb2-a34a-84d91352b2df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:02.516564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2ae3a6-fe80-4bb2-a34a-84d91352b2df admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:02.516845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2ae3a6-fe80-4bb2-a34a-84d91352b2df admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:02.517164 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2ae3a6-fe80-4bb2-a34a-84d91352b2df admin admin] PATH_INFO: `/v3/projects/e66308011618464f9e6519da298e7d71/users/6113a6d40c0047e1b1ed2849ac6349b4/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:02.519154 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2ae3a6-fe80-4bb2-a34a-84d91352b2df admin admin] RBAC: Authorizing `identity:create_grant(project_id=e66308011618464f9e6519da298e7d71, user_id=6113a6d40c0047e1b1ed2849ac6349b4, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:02.522867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2ae3a6-fe80-4bb2-a34a-84d91352b2df admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:02.526282 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:02.533286 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1300/2611] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:13:02 2022] PUT /identity/v3/projects/e66308011618464f9e6519da298e7d71/users/6113a6d40c0047e1b1ed2849ac6349b4/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:02.534654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21603c1d-e1aa-4750-98b1-bcfcf45c99a0 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:02.537090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21603c1d-e1aa-4750-98b1-bcfcf45c99a0 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:02.537090 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21603c1d-e1aa-4750-98b1-bcfcf45c99a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:02.556383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:02.564924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-21603c1d-e1aa-4750-98b1-bcfcf45c99a0 None None] MFA Rules not processed for user `2c1fe639fb5847fe9bbe1303b8688215`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:02.566189 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-21603c1d-e1aa-4750-98b1-bcfcf45c99a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:02.588748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1312/2612] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:13:02 2022] POST /identity/v3/projects => generated 379 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:02.590218 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfa7da8-fe6a-4169-b1c0-b455b6fb30dc None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:02.590604 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfa7da8-fe6a-4169-b1c0-b455b6fb30dc None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:02.590980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfa7da8-fe6a-4169-b1c0-b455b6fb30dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:02.620265 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-fdfa7da8-fe6a-4169-b1c0-b455b6fb30dc None None] MFA Rules not processed for user `6113a6d40c0047e1b1ed2849ac6349b4`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:02.621550 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-fdfa7da8-fe6a-4169-b1c0-b455b6fb30dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:02.634094 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1301/2613] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:02 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:02.635406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21603c1d-e1aa-4750-98b1-bcfcf45c99a0 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:02.696976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1313/2614] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:02 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:02.697918 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfa7da8-fe6a-4169-b1c0-b455b6fb30dc None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:02.714061 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21603c1d-e1aa-4750-98b1-bcfcf45c99a0 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:02.764222 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfa7da8-fe6a-4169-b1c0-b455b6fb30dc None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:02.769185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebebd5b-e794-43eb-ad7a-06a04adf0553 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:02.770421 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebebd5b-e794-43eb-ad7a-06a04adf0553 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:02.770715 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebebd5b-e794-43eb-ad7a-06a04adf0553 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:02.770986 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebebd5b-e794-43eb-ad7a-06a04adf0553 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:02.771312 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f898a482-99e3-4319-a767-cf472a2bfc54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:02.771613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebebd5b-e794-43eb-ad7a-06a04adf0553 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:02.771936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f898a482-99e3-4319-a767-cf472a2bfc54 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:02.772210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f898a482-99e3-4319-a767-cf472a2bfc54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:02.772485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f898a482-99e3-4319-a767-cf472a2bfc54 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:02.773143 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f898a482-99e3-4319-a767-cf472a2bfc54 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:02.773574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebebd5b-e794-43eb-ad7a-06a04adf0553 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:02.775074 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f898a482-99e3-4319-a767-cf472a2bfc54 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:02.853956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1302/2615] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:13:02 2022] POST /identity/v3/users => generated 290 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:02.854917 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f898a482-99e3-4319-a767-cf472a2bfc54 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:02.856520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f898a482-99e3-4319-a767-cf472a2bfc54 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:02.893767 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1314/2616] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:13:02 2022] POST /identity/v3/projects => generated 373 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:02.893767 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebebd5b-e794-43eb-ad7a-06a04adf0553 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:02.901052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:02.920229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f898a482-99e3-4319-a767-cf472a2bfc54 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:02.977500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce861c22-4caf-4313-9439-943f5efc2324 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:02.978141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce861c22-4caf-4313-9439-943f5efc2324 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:02.978654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce861c22-4caf-4313-9439-943f5efc2324 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:02.978838 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce861c22-4caf-4313-9439-943f5efc2324 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:03.046352 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce861c22-4caf-4313-9439-943f5efc2324 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:03.047796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce861c22-4caf-4313-9439-943f5efc2324 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:03.057382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1303/2617] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:02 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:13:03.058165 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce861c22-4caf-4313-9439-943f5efc2324 service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:03.081043 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebebd5b-e794-43eb-ad7a-06a04adf0553 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:03.089276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4572b745-0b8d-43c7-adc7-5a8df007ddd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:03.089276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4572b745-0b8d-43c7-adc7-5a8df007ddd7 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:03.089276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4572b745-0b8d-43c7-adc7-5a8df007ddd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:03.089276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4572b745-0b8d-43c7-adc7-5a8df007ddd7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:03.089694 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4572b745-0b8d-43c7-adc7-5a8df007ddd7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:03.090060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4572b745-0b8d-43c7-adc7-5a8df007ddd7 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:03.093282 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:03.101748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1315/2618] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:02 2022] GET /identity/v3/roles => generated 1420 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:03.101748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4572b745-0b8d-43c7-adc7-5a8df007ddd7 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:03.112184 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4572b745-0b8d-43c7-adc7-5a8df007ddd7 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:03.117742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e2ad8b-cffa-45a0-8d4f-3d62f9f64e6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:03.118340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e2ad8b-cffa-45a0-8d4f-3d62f9f64e6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:03.118614 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e2ad8b-cffa-45a0-8d4f-3d62f9f64e6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:03.118887 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e2ad8b-cffa-45a0-8d4f-3d62f9f64e6a admin admin] PATH_INFO: `/v3/users/f9e3043364ef49ca94e41297bd02987f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:03.119999 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e2ad8b-cffa-45a0-8d4f-3d62f9f64e6a admin admin] RBAC: Authorizing `identity:delete_user(user_id=f9e3043364ef49ca94e41297bd02987f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:03.121921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e2ad8b-cffa-45a0-8d4f-3d62f9f64e6a admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:03.139449 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce861c22-4caf-4313-9439-943f5efc2324 service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:03.144126 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ced860c-2545-47f0-8675-891deb2605e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:03.145378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ced860c-2545-47f0-8675-891deb2605e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:03.145651 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ced860c-2545-47f0-8675-891deb2605e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:03.145928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ced860c-2545-47f0-8675-891deb2605e0 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:03.146560 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ced860c-2545-47f0-8675-891deb2605e0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:03.148448 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ced860c-2545-47f0-8675-891deb2605e0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:03.254462 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1304/2619] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:13:03 2022] POST /identity/v3/users => generated 293 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:03.255312 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ced860c-2545-47f0-8675-891deb2605e0 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:03.264616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ced860c-2545-47f0-8675-891deb2605e0 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:03.269317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cababe22-0f52-4088-9ce6-45d091ee84ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:03.270669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cababe22-0f52-4088-9ce6-45d091ee84ad admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:03.270950 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cababe22-0f52-4088-9ce6-45d091ee84ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:03.271221 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cababe22-0f52-4088-9ce6-45d091ee84ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:03.271900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cababe22-0f52-4088-9ce6-45d091ee84ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:03.273516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cababe22-0f52-4088-9ce6-45d091ee84ad admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:03.283634 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1305/2620] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:03 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:03.284408 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cababe22-0f52-4088-9ce6-45d091ee84ad admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:03.294665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cababe22-0f52-4088-9ce6-45d091ee84ad admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:03.299254 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b748cf89-5b7f-4dc4-8b74-1975c1eb6f25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:03.299814 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b748cf89-5b7f-4dc4-8b74-1975c1eb6f25 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:03.300092 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b748cf89-5b7f-4dc4-8b74-1975c1eb6f25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:03.300388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b748cf89-5b7f-4dc4-8b74-1975c1eb6f25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:03.301058 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b748cf89-5b7f-4dc4-8b74-1975c1eb6f25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:03.302617 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b748cf89-5b7f-4dc4-8b74-1975c1eb6f25 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:03.313129 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1306/2621] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:03 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:03.313907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b748cf89-5b7f-4dc4-8b74-1975c1eb6f25 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:03.314998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-c7e2ad8b-cffa-45a0-8d4f-3d62f9f64e6a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f9e3043364ef49ca94e41297bd02987f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:03.324894 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b748cf89-5b7f-4dc4-8b74-1975c1eb6f25 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:03.329619 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878d7826-29b1-4d52-a472-a869b8855947 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:03.330248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878d7826-29b1-4d52-a472-a869b8855947 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:03.330526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878d7826-29b1-4d52-a472-a869b8855947 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:03.330797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878d7826-29b1-4d52-a472-a869b8855947 admin admin] PATH_INFO: `/v3/projects/6841cb8436ae43ff843cf938e91711ab/users/c106b19876d5474e94281cd06ee7316a/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:03.336148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:03.345767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878d7826-29b1-4d52-a472-a869b8855947 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6841cb8436ae43ff843cf938e91711ab, user_id=c106b19876d5474e94281cd06ee7316a, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:03.349248 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878d7826-29b1-4d52-a472-a869b8855947 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:03.385656 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-c7e2ad8b-cffa-45a0-8d4f-3d62f9f64e6a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f9e3043364ef49ca94e41297bd02987f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:03.398684 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-c7e2ad8b-cffa-45a0-8d4f-3d62f9f64e6a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f9e3043364ef49ca94e41297bd02987f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:03.399284 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-c7e2ad8b-cffa-45a0-8d4f-3d62f9f64e6a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f9e3043364ef49ca94e41297bd02987f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:03.406295 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-c7e2ad8b-cffa-45a0-8d4f-3d62f9f64e6a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f9e3043364ef49ca94e41297bd02987f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:03.413203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1316/2622] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:13:03 2022] DELETE /identity/v3/users/f9e3043364ef49ca94e41297bd02987f => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:03.414161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e2ad8b-cffa-45a0-8d4f-3d62f9f64e6a admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:03.415497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c7e2ad8b-cffa-45a0-8d4f-3d62f9f64e6a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:03.423069 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1307/2623] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:13:03 2022] PUT /identity/v3/projects/6841cb8436ae43ff843cf938e91711ab/users/c106b19876d5474e94281cd06ee7316a/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:03.424764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cd3dfc-7dc2-41e1-ad7e-15617d7c8f60 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:03.425050 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cd3dfc-7dc2-41e1-ad7e-15617d7c8f60 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:03.425342 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cd3dfc-7dc2-41e1-ad7e-15617d7c8f60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:03.447669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-c8cd3dfc-7dc2-41e1-ad7e-15617d7c8f60 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:03.448988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-c8cd3dfc-7dc2-41e1-ad7e-15617d7c8f60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:03.508940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e2ad8b-cffa-45a0-8d4f-3d62f9f64e6a admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:03.517520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1308/2624] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:03.518799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d592dc-3a40-4a65-a97c-eb18933f7fb9 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:03.525234 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d592dc-3a40-4a65-a97c-eb18933f7fb9 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:03.525234 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d592dc-3a40-4a65-a97c-eb18933f7fb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:03.548355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-a9d592dc-3a40-4a65-a97c-eb18933f7fb9 None None] MFA Rules not processed for user `c106b19876d5474e94281cd06ee7316a`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:03.549605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a9d592dc-3a40-4a65-a97c-eb18933f7fb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:03.565986 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175655ef-e05a-4c6f-9198-22e7c8fcc786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:03.567307 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175655ef-e05a-4c6f-9198-22e7c8fcc786 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:03.567583 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175655ef-e05a-4c6f-9198-22e7c8fcc786 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:03.567854 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175655ef-e05a-4c6f-9198-22e7c8fcc786 admin admin] PATH_INFO: `/v3/projects/da3cd3ece2bf4885892aa6b2afdbd787/users/cddb4a0de03e4d369cbd1428de5c14f8/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:03.583125 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175655ef-e05a-4c6f-9198-22e7c8fcc786 admin admin] RBAC: Authorizing `identity:create_grant(project_id=da3cd3ece2bf4885892aa6b2afdbd787, user_id=cddb4a0de03e4d369cbd1428de5c14f8, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:03.585920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175655ef-e05a-4c6f-9198-22e7c8fcc786 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:03.614265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1309/2625] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:03 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:03.615170 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d592dc-3a40-4a65-a97c-eb18933f7fb9 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:03.616441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a9d592dc-3a40-4a65-a97c-eb18933f7fb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:03.655749 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:03.660887 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1317/2626] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:13:03 2022] PUT /identity/v3/projects/da3cd3ece2bf4885892aa6b2afdbd787/users/cddb4a0de03e4d369cbd1428de5c14f8/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:03.664135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175655ef-e05a-4c6f-9198-22e7c8fcc786 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:03.665105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-175655ef-e05a-4c6f-9198-22e7c8fcc786 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:03.673223 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d592dc-3a40-4a65-a97c-eb18933f7fb9 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:03.702378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:03.723809 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23208ca0-38c9-4ffb-a34e-5bd4a34aeb9f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:03.724415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23208ca0-38c9-4ffb-a34e-5bd4a34aeb9f service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:03.724693 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23208ca0-38c9-4ffb-a34e-5bd4a34aeb9f service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:03.724966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23208ca0-38c9-4ffb-a34e-5bd4a34aeb9f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:03.729513 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175655ef-e05a-4c6f-9198-22e7c8fcc786 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:03.734276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b721da4b-f25d-45d5-8326-1f4ff6484cd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:03.734880 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b721da4b-f25d-45d5-8326-1f4ff6484cd0 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:03.735161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b721da4b-f25d-45d5-8326-1f4ff6484cd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:03.735275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23208ca0-38c9-4ffb-a34e-5bd4a34aeb9f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:03.735439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b721da4b-f25d-45d5-8326-1f4ff6484cd0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:03.736069 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b721da4b-f25d-45d5-8326-1f4ff6484cd0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:03.736731 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23208ca0-38c9-4ffb-a34e-5bd4a34aeb9f service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:03.738008 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b721da4b-f25d-45d5-8326-1f4ff6484cd0 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:03.746885 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1310/2627] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:13:03.747914 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23208ca0-38c9-4ffb-a34e-5bd4a34aeb9f service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:03.758074 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23208ca0-38c9-4ffb-a34e-5bd4a34aeb9f service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:03.762845 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5856af9-aac0-4ab6-8e05-d38175958c8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:03.763443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5856af9-aac0-4ab6-8e05-d38175958c8e admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:03.763730 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5856af9-aac0-4ab6-8e05-d38175958c8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:03.764003 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5856af9-aac0-4ab6-8e05-d38175958c8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:03.764674 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5856af9-aac0-4ab6-8e05-d38175958c8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:03.766299 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5856af9-aac0-4ab6-8e05-d38175958c8e admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:03.777314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1311/2628] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:03.781391 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5856af9-aac0-4ab6-8e05-d38175958c8e admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:03.791322 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5856af9-aac0-4ab6-8e05-d38175958c8e admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:03.795963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65c31a7-c474-408c-930e-f28ae71083e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:03.796903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65c31a7-c474-408c-930e-f28ae71083e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:03.797211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65c31a7-c474-408c-930e-f28ae71083e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:03.797489 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65c31a7-c474-408c-930e-f28ae71083e3 admin admin] PATH_INFO: `/v3/projects/da3cd3ece2bf4885892aa6b2afdbd787/users/cddb4a0de03e4d369cbd1428de5c14f8/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:03.799382 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65c31a7-c474-408c-930e-f28ae71083e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=da3cd3ece2bf4885892aa6b2afdbd787, user_id=cddb4a0de03e4d369cbd1428de5c14f8, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:03.802591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65c31a7-c474-408c-930e-f28ae71083e3 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:03.825162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1312/2629] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:13:03 2022] PUT /identity/v3/projects/da3cd3ece2bf4885892aa6b2afdbd787/users/cddb4a0de03e4d369cbd1428de5c14f8/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:03.829428 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78673652-3b35-428e-8c60-1219ef0625fb None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:03.829738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78673652-3b35-428e-8c60-1219ef0625fb None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:03.830033 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78673652-3b35-428e-8c60-1219ef0625fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:03.834463 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1318/2630] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:13:03 2022] POST /identity/v3/projects => generated 379 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:03.835581 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b721da4b-f25d-45d5-8326-1f4ff6484cd0 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:03.862510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-78673652-3b35-428e-8c60-1219ef0625fb None None] MFA Rules not processed for user `cddb4a0de03e4d369cbd1428de5c14f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:03.866120 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-78673652-3b35-428e-8c60-1219ef0625fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:03.902481 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b721da4b-f25d-45d5-8326-1f4ff6484cd0 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:03.911920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e768e17f-e488-4235-8912-91e50056027d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:03.911920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e768e17f-e488-4235-8912-91e50056027d admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:03.911920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e768e17f-e488-4235-8912-91e50056027d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:03.911920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e768e17f-e488-4235-8912-91e50056027d admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:03.912385 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e768e17f-e488-4235-8912-91e50056027d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:03.913108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e768e17f-e488-4235-8912-91e50056027d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:03.942017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1313/2631] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:03 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:13:03.956790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78673652-3b35-428e-8c60-1219ef0625fb None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:03.959110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-78673652-3b35-428e-8c60-1219ef0625fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:03.993777 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1319/2632] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:13:03 2022] POST /identity/v3/users => generated 290 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:03.995244 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e768e17f-e488-4235-8912-91e50056027d admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:04.009938 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e768e17f-e488-4235-8912-91e50056027d admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:04.016875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f43f21d-b1b7-47ab-8860-1a3154fb1625 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:04.016875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f43f21d-b1b7-47ab-8860-1a3154fb1625 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:04.016875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f43f21d-b1b7-47ab-8860-1a3154fb1625 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:04.016875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f43f21d-b1b7-47ab-8860-1a3154fb1625 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:04.017321 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f43f21d-b1b7-47ab-8860-1a3154fb1625 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:04.018376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f43f21d-b1b7-47ab-8860-1a3154fb1625 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:04.025028 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78673652-3b35-428e-8c60-1219ef0625fb None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:04.029212 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3f45b5-2408-46c2-a112-1b73563ea3e7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:04.029983 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3f45b5-2408-46c2-a112-1b73563ea3e7 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:04.030140 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3f45b5-2408-46c2-a112-1b73563ea3e7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:04.030397 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3f45b5-2408-46c2-a112-1b73563ea3e7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:04.031856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1320/2633] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:03 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:04.035761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f43f21d-b1b7-47ab-8860-1a3154fb1625 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:04.041533 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3f45b5-2408-46c2-a112-1b73563ea3e7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:04.043114 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3f45b5-2408-46c2-a112-1b73563ea3e7 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:04.046155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f43f21d-b1b7-47ab-8860-1a3154fb1625 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:04.051593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72359441-1480-4173-80b7-66437a26a6c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:04.052284 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72359441-1480-4173-80b7-66437a26a6c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:04.052587 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72359441-1480-4173-80b7-66437a26a6c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:04.053260 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72359441-1480-4173-80b7-66437a26a6c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:04.053667 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72359441-1480-4173-80b7-66437a26a6c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:04.055319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1314/2634] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:03 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:13:04.055474 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72359441-1480-4173-80b7-66437a26a6c3 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:04.067254 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1321/2635] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:04 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:13:04.075457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3f45b5-2408-46c2-a112-1b73563ea3e7 service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:04.088244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3f45b5-2408-46c2-a112-1b73563ea3e7 service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:04.092909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb061f6d-db8b-451e-a45e-dd49b828596b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:04.093588 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb061f6d-db8b-451e-a45e-dd49b828596b admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:04.093869 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb061f6d-db8b-451e-a45e-dd49b828596b admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:04.094212 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb061f6d-db8b-451e-a45e-dd49b828596b admin admin] PATH_INFO: `/v3/projects/7ca56a8fae1747c6b2a17b632cb8d430/users/51a68bcdc8974e9c9ded50b0c0975157/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:04.110798 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb061f6d-db8b-451e-a45e-dd49b828596b admin admin] RBAC: Authorizing `identity:create_grant(project_id=7ca56a8fae1747c6b2a17b632cb8d430, user_id=51a68bcdc8974e9c9ded50b0c0975157, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:04.114934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb061f6d-db8b-451e-a45e-dd49b828596b admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:04.178009 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72359441-1480-4173-80b7-66437a26a6c3 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:04.179736 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1315/2636] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:13:04 2022] PUT /identity/v3/projects/7ca56a8fae1747c6b2a17b632cb8d430/users/51a68bcdc8974e9c9ded50b0c0975157/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:13:04.182926 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-72359441-1480-4173-80b7-66437a26a6c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:04.190195 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e1bad2-90c9-4974-abd8-6e46a8407fbb None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:04.190485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e1bad2-90c9-4974-abd8-6e46a8407fbb None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:04.190762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e1bad2-90c9-4974-abd8-6e46a8407fbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:04.211208 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72359441-1480-4173-80b7-66437a26a6c3 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:04.221615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:04.225790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-78e1bad2-90c9-4974-abd8-6e46a8407fbb None None] MFA Rules not processed for user `51a68bcdc8974e9c9ded50b0c0975157`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:04.227074 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-78e1bad2-90c9-4974-abd8-6e46a8407fbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:04.266850 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:04.275744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11274119-4d41-47db-85ce-6a3d3f13d472 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:04.276411 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11274119-4d41-47db-85ce-6a3d3f13d472 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:04.276688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11274119-4d41-47db-85ce-6a3d3f13d472 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:04.276956 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11274119-4d41-47db-85ce-6a3d3f13d472 admin admin] PATH_INFO: `/v3/projects/c42193bc84da4389a3c050979cd759e3` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:04.278262 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11274119-4d41-47db-85ce-6a3d3f13d472 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c42193bc84da4389a3c050979cd759e3)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:04.280141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11274119-4d41-47db-85ce-6a3d3f13d472 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:04.295118 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1316/2637] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:04 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:04.305546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e1bad2-90c9-4974-abd8-6e46a8407fbb None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:04.318392 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e1bad2-90c9-4974-abd8-6e46a8407fbb None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:04.323163 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a88bb83-3a21-4de0-9e43-a15866e99200 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:04.323756 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a88bb83-3a21-4de0-9e43-a15866e99200 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:04.324035 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a88bb83-3a21-4de0-9e43-a15866e99200 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:04.324307 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a88bb83-3a21-4de0-9e43-a15866e99200 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:04.324937 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a88bb83-3a21-4de0-9e43-a15866e99200 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:04.326992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a88bb83-3a21-4de0-9e43-a15866e99200 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:04.402632 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-11274119-4d41-47db-85ce-6a3d3f13d472 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c42193bc84da4389a3c050979cd759e3', 'request_id': 'req-11274119-4d41-47db-85ce-6a3d3f13d472'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:04.408257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-11274119-4d41-47db-85ce-6a3d3f13d472 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c42193bc84da4389a3c050979cd759e3', 'request_id': 'req-11274119-4d41-47db-85ce-6a3d3f13d472'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:04.415144 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-11274119-4d41-47db-85ce-6a3d3f13d472 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c42193bc84da4389a3c050979cd759e3', 'request_id': 'req-11274119-4d41-47db-85ce-6a3d3f13d472'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:04.439181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1317/2638] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:13:04 2022] POST /identity/v3/projects => generated 377 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:04.440084 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a88bb83-3a21-4de0-9e43-a15866e99200 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:04.477195 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-11274119-4d41-47db-85ce-6a3d3f13d472 admin admin] The token cache is being invalidate because project c42193bc84da4389a3c050979cd759e3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:13:04.477616 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-11274119-4d41-47db-85ce-6a3d3f13d472 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:04.478388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-11274119-4d41-47db-85ce-6a3d3f13d472 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:04.483710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1322/2639] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:13:04 2022] DELETE /identity/v3/projects/c42193bc84da4389a3c050979cd759e3 => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:04.483710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11274119-4d41-47db-85ce-6a3d3f13d472 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:04.483710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-11274119-4d41-47db-85ce-6a3d3f13d472 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:04.500087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a88bb83-3a21-4de0-9e43-a15866e99200 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:04.503990 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671868c3-d00c-4f0b-923e-1914ba2b8760 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:04.504771 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671868c3-d00c-4f0b-923e-1914ba2b8760 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:04.505057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671868c3-d00c-4f0b-923e-1914ba2b8760 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:04.505398 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671868c3-d00c-4f0b-923e-1914ba2b8760 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:04.507163 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-671868c3-d00c-4f0b-923e-1914ba2b8760 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:04.549104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11274119-4d41-47db-85ce-6a3d3f13d472 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:04.570836 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-671868c3-d00c-4f0b-923e-1914ba2b8760 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:04.572262 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-671868c3-d00c-4f0b-923e-1914ba2b8760 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:04.581813 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1318/2640] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:04 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:13:04.583242 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671868c3-d00c-4f0b-923e-1914ba2b8760 service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:04.585110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-671868c3-d00c-4f0b-923e-1914ba2b8760 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:04.610963 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d27caa-0ad6-48b3-84db-144ce54a20c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:04.611628 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d27caa-0ad6-48b3-84db-144ce54a20c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:04.611902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d27caa-0ad6-48b3-84db-144ce54a20c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:04.612168 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d27caa-0ad6-48b3-84db-144ce54a20c4 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:04.612796 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d27caa-0ad6-48b3-84db-144ce54a20c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:04.614903 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d27caa-0ad6-48b3-84db-144ce54a20c4 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:04.630503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:04.655606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671868c3-d00c-4f0b-923e-1914ba2b8760 service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:04.662187 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a996ac7f-80e2-4c5f-b5a2-845236026528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:04.663849 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a996ac7f-80e2-4c5f-b5a2-845236026528 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:04.665611 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a996ac7f-80e2-4c5f-b5a2-845236026528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:04.665906 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a996ac7f-80e2-4c5f-b5a2-845236026528 admin admin] PATH_INFO: `/v3/users/6113a6d40c0047e1b1ed2849ac6349b4` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:04.668285 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a996ac7f-80e2-4c5f-b5a2-845236026528 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6113a6d40c0047e1b1ed2849ac6349b4)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:04.670350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a996ac7f-80e2-4c5f-b5a2-845236026528 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:04.700271 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1323/2641] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:13:04 2022] POST /identity/v3/users => generated 295 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:04.704185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d27caa-0ad6-48b3-84db-144ce54a20c4 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:04.714493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d27caa-0ad6-48b3-84db-144ce54a20c4 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:04.719043 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dff4015-7f8f-4bf6-8651-6ac10015515e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:04.719625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dff4015-7f8f-4bf6-8651-6ac10015515e admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:04.719901 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dff4015-7f8f-4bf6-8651-6ac10015515e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:04.720171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dff4015-7f8f-4bf6-8651-6ac10015515e admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:04.720806 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dff4015-7f8f-4bf6-8651-6ac10015515e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:04.722431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dff4015-7f8f-4bf6-8651-6ac10015515e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:04.734053 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1324/2642] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:13:04.740095 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dff4015-7f8f-4bf6-8651-6ac10015515e admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:04.750497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dff4015-7f8f-4bf6-8651-6ac10015515e admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:04.755620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6965791e-7ec1-4643-8069-b0010df16500 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:04.756257 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6965791e-7ec1-4643-8069-b0010df16500 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:04.756537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6965791e-7ec1-4643-8069-b0010df16500 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:04.756810 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6965791e-7ec1-4643-8069-b0010df16500 admin admin] PATH_INFO: `/v3/projects/136206f1ccd544a5b27352822a9de9fc/users/3c8ab6c7e08349b699c7ee9366b1c969/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:04.759921 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:04.772875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6965791e-7ec1-4643-8069-b0010df16500 admin admin] RBAC: Authorizing `identity:create_grant(project_id=136206f1ccd544a5b27352822a9de9fc, user_id=3c8ab6c7e08349b699c7ee9366b1c969, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:04.775555 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6965791e-7ec1-4643-8069-b0010df16500 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:04.851148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a996ac7f-80e2-4c5f-b5a2-845236026528 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6113a6d40c0047e1b1ed2849ac6349b4'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:04.851832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a996ac7f-80e2-4c5f-b5a2-845236026528 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6113a6d40c0047e1b1ed2849ac6349b4'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:04.872362 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1325/2643] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:13:04 2022] PUT /identity/v3/projects/136206f1ccd544a5b27352822a9de9fc/users/3c8ab6c7e08349b699c7ee9366b1c969/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:04.876591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6965791e-7ec1-4643-8069-b0010df16500 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:04.878306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-6965791e-7ec1-4643-8069-b0010df16500 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:04.912314 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a996ac7f-80e2-4c5f-b5a2-845236026528 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6113a6d40c0047e1b1ed2849ac6349b4'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:04.927153 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a996ac7f-80e2-4c5f-b5a2-845236026528 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6113a6d40c0047e1b1ed2849ac6349b4'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:04.927972 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a996ac7f-80e2-4c5f-b5a2-845236026528 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6113a6d40c0047e1b1ed2849ac6349b4'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:04.940704 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1319/2644] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:13:04 2022] DELETE /identity/v3/users/6113a6d40c0047e1b1ed2849ac6349b4 => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:04.948111 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6965791e-7ec1-4643-8069-b0010df16500 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:05.029303 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b82bf31-e03f-4d1b-9558-94f6fffbe6fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:05.030368 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b82bf31-e03f-4d1b-9558-94f6fffbe6fb admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:05.030661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b82bf31-e03f-4d1b-9558-94f6fffbe6fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:05.030939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b82bf31-e03f-4d1b-9558-94f6fffbe6fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:05.031653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b82bf31-e03f-4d1b-9558-94f6fffbe6fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:05.033815 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b82bf31-e03f-4d1b-9558-94f6fffbe6fb admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:05.046660 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1326/2645] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:04 2022] GET /identity/v3/roles => generated 1420 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:13:05.051299 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a996ac7f-80e2-4c5f-b5a2-845236026528 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:05.061472 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a996ac7f-80e2-4c5f-b5a2-845236026528 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:05.066203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a434ee04-23e2-4ac3-89a8-340c1b6df29c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:05.066899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a434ee04-23e2-4ac3-89a8-340c1b6df29c admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:05.067257 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a434ee04-23e2-4ac3-89a8-340c1b6df29c admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:05.067572 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a434ee04-23e2-4ac3-89a8-340c1b6df29c admin admin] PATH_INFO: `/v3/projects/136206f1ccd544a5b27352822a9de9fc/users/3c8ab6c7e08349b699c7ee9366b1c969/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:05.069541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a434ee04-23e2-4ac3-89a8-340c1b6df29c admin admin] RBAC: Authorizing `identity:create_grant(project_id=136206f1ccd544a5b27352822a9de9fc, user_id=3c8ab6c7e08349b699c7ee9366b1c969, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:05.072904 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a434ee04-23e2-4ac3-89a8-340c1b6df29c admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:05.083695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1320/2646] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:13:05 2022] PUT /identity/v3/projects/136206f1ccd544a5b27352822a9de9fc/users/3c8ab6c7e08349b699c7ee9366b1c969/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:05.096503 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985fe849-3bcf-43ae-bb9d-990b64b51af3 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:05.101659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985fe849-3bcf-43ae-bb9d-990b64b51af3 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:05.101659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985fe849-3bcf-43ae-bb9d-990b64b51af3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:05.126617 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-985fe849-3bcf-43ae-bb9d-990b64b51af3 None None] MFA Rules not processed for user `3c8ab6c7e08349b699c7ee9366b1c969`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:05.127841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-985fe849-3bcf-43ae-bb9d-990b64b51af3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:05.199038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1327/2647] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:05 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:05.210312 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a434ee04-23e2-4ac3-89a8-340c1b6df29c admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:05.212308 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a434ee04-23e2-4ac3-89a8-340c1b6df29c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:05.274258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a434ee04-23e2-4ac3-89a8-340c1b6df29c admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:05.278209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f1b2a9-90f0-4518-9a40-678ac7b2ecb6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:05.278820 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f1b2a9-90f0-4518-9a40-678ac7b2ecb6 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:05.279103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f1b2a9-90f0-4518-9a40-678ac7b2ecb6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:05.279378 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f1b2a9-90f0-4518-9a40-678ac7b2ecb6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:05.289684 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f1b2a9-90f0-4518-9a40-678ac7b2ecb6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:05.291041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f1b2a9-90f0-4518-9a40-678ac7b2ecb6 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:05.301581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1321/2648] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:05 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:13:05.400074 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985fe849-3bcf-43ae-bb9d-990b64b51af3 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:05.401689 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-985fe849-3bcf-43ae-bb9d-990b64b51af3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:05.415619 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f1b2a9-90f0-4518-9a40-678ac7b2ecb6 service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:05.417162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b3f1b2a9-90f0-4518-9a40-678ac7b2ecb6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:05.429892 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f1b2a9-90f0-4518-9a40-678ac7b2ecb6 service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:05.433762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6484a283-0eda-4ee1-97f2-d3a15768d628 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:05.434795 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6484a283-0eda-4ee1-97f2-d3a15768d628 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:05.434967 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6484a283-0eda-4ee1-97f2-d3a15768d628 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:05.435210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6484a283-0eda-4ee1-97f2-d3a15768d628 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:05.437288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6484a283-0eda-4ee1-97f2-d3a15768d628 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:05.484136 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985fe849-3bcf-43ae-bb9d-990b64b51af3 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:05.487902 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:05.489986 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ae7b1e-9c27-453b-9b3d-ec1855beb52a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:05.490761 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ae7b1e-9c27-453b-9b3d-ec1855beb52a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:05.491136 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ae7b1e-9c27-453b-9b3d-ec1855beb52a admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:05.491473 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ae7b1e-9c27-453b-9b3d-ec1855beb52a admin admin] PATH_INFO: `/v3/projects/e66308011618464f9e6519da298e7d71` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:05.492863 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ae7b1e-9c27-453b-9b3d-ec1855beb52a admin admin] RBAC: Authorizing `identity:delete_project(project_id=e66308011618464f9e6519da298e7d71)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:05.494943 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ae7b1e-9c27-453b-9b3d-ec1855beb52a admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:05.499788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:05.504681 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6484a283-0eda-4ee1-97f2-d3a15768d628 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:05.506145 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6484a283-0eda-4ee1-97f2-d3a15768d628 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:05.515636 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1322/2649] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:05 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:13:05.592860 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-04ae7b1e-9c27-453b-9b3d-ec1855beb52a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e66308011618464f9e6519da298e7d71', 'request_id': 'req-04ae7b1e-9c27-453b-9b3d-ec1855beb52a'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:05.599068 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-04ae7b1e-9c27-453b-9b3d-ec1855beb52a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e66308011618464f9e6519da298e7d71', 'request_id': 'req-04ae7b1e-9c27-453b-9b3d-ec1855beb52a'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:05.605405 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-04ae7b1e-9c27-453b-9b3d-ec1855beb52a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e66308011618464f9e6519da298e7d71', 'request_id': 'req-04ae7b1e-9c27-453b-9b3d-ec1855beb52a'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:05.700002 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-04ae7b1e-9c27-453b-9b3d-ec1855beb52a admin admin] The token cache is being invalidate because project e66308011618464f9e6519da298e7d71 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:13:05.700384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-04ae7b1e-9c27-453b-9b3d-ec1855beb52a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:05.701046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-04ae7b1e-9c27-453b-9b3d-ec1855beb52a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:05.702555 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1328/2650] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:13:05 2022] DELETE /identity/v3/projects/e66308011618464f9e6519da298e7d71 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:05.709044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e00d71-a359-481a-a6e4-6a4f855d003d None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:05.709429 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e00d71-a359-481a-a6e4-6a4f855d003d None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:05.709781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e00d71-a359-481a-a6e4-6a4f855d003d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:05.729970 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-51e00d71-a359-481a-a6e4-6a4f855d003d None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:05.731104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-51e00d71-a359-481a-a6e4-6a4f855d003d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:05.792156 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1323/2651] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:05.800158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79984ba4-61e7-4df0-a00c-5418e77081cc None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:05.800440 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79984ba4-61e7-4df0-a00c-5418e77081cc None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:05.801438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79984ba4-61e7-4df0-a00c-5418e77081cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:05.821186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-79984ba4-61e7-4df0-a00c-5418e77081cc None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:05.822401 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-79984ba4-61e7-4df0-a00c-5418e77081cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:05.830350 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1329/2652] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:05.834707 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e00d71-a359-481a-a6e4-6a4f855d003d None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:05.844774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e00d71-a359-481a-a6e4-6a4f855d003d None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:05.850338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d348e2a-c06b-402a-8f6b-0b6d1975ada3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:05.850949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d348e2a-c06b-402a-8f6b-0b6d1975ada3 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:05.851255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d348e2a-c06b-402a-8f6b-0b6d1975ada3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:05.851527 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d348e2a-c06b-402a-8f6b-0b6d1975ada3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:05.852178 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d348e2a-c06b-402a-8f6b-0b6d1975ada3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:13:05.852488 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d348e2a-c06b-402a-8f6b-0b6d1975ada3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:05.854110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d348e2a-c06b-402a-8f6b-0b6d1975ada3 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:05.866294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1324/2653] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:13:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:13:05.871787 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79984ba4-61e7-4df0-a00c-5418e77081cc None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:05.881327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79984ba4-61e7-4df0-a00c-5418e77081cc None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:05.886428 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94e3e3b-b40c-4301-976f-59cd5d53ada8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:05.887088 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94e3e3b-b40c-4301-976f-59cd5d53ada8 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:05.887383 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94e3e3b-b40c-4301-976f-59cd5d53ada8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:05.887860 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94e3e3b-b40c-4301-976f-59cd5d53ada8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:05.889194 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94e3e3b-b40c-4301-976f-59cd5d53ada8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:05.894729 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94e3e3b-b40c-4301-976f-59cd5d53ada8 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:05.976555 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1330/2654] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:13:05 2022] POST /identity/v3/projects => generated 381 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:05.981041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d348e2a-c06b-402a-8f6b-0b6d1975ada3 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:05.992465 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d348e2a-c06b-402a-8f6b-0b6d1975ada3 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:06.049918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be65b560-4946-4d5b-a91f-087f09351be7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:06.050612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be65b560-4946-4d5b-a91f-087f09351be7 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:06.051423 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be65b560-4946-4d5b-a91f-087f09351be7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:06.051759 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be65b560-4946-4d5b-a91f-087f09351be7 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:06.052389 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be65b560-4946-4d5b-a91f-087f09351be7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:06.054381 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be65b560-4946-4d5b-a91f-087f09351be7 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:06.158266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1325/2655] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:13:05 2022] POST /identity/v3/users => generated 291 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:06.163031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94e3e3b-b40c-4301-976f-59cd5d53ada8 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:06.173112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94e3e3b-b40c-4301-976f-59cd5d53ada8 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:06.177862 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137c5303-5f2c-42aa-bc46-cd84f627cc0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:06.178496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137c5303-5f2c-42aa-bc46-cd84f627cc0c admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:06.178772 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137c5303-5f2c-42aa-bc46-cd84f627cc0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:06.179057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137c5303-5f2c-42aa-bc46-cd84f627cc0c admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:06.179678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137c5303-5f2c-42aa-bc46-cd84f627cc0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:06.181289 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137c5303-5f2c-42aa-bc46-cd84f627cc0c admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:06.192005 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1331/2656] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:06 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:06.198883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be65b560-4946-4d5b-a91f-087f09351be7 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:06.210386 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be65b560-4946-4d5b-a91f-087f09351be7 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:06.215552 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ad340f-d908-4b48-b98b-780757dcda83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:06.224865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ad340f-d908-4b48-b98b-780757dcda83 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:06.224865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ad340f-d908-4b48-b98b-780757dcda83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:06.224865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ad340f-d908-4b48-b98b-780757dcda83 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:06.224865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ad340f-d908-4b48-b98b-780757dcda83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:06.224865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ad340f-d908-4b48-b98b-780757dcda83 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:06.230948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1326/2657] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:06 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:06.235369 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137c5303-5f2c-42aa-bc46-cd84f627cc0c admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:06.245521 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137c5303-5f2c-42aa-bc46-cd84f627cc0c admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:06.250399 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8349d50c-7dce-4604-b9a0-c99196165b3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:06.251073 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8349d50c-7dce-4604-b9a0-c99196165b3d admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:06.251350 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8349d50c-7dce-4604-b9a0-c99196165b3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:06.251625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8349d50c-7dce-4604-b9a0-c99196165b3d admin admin] PATH_INFO: `/v3/projects/063c97af8ef74f7f86873dfa47c6798c/users/995b47fae29e4f49ba05ae3858b543c4/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:06.266772 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8349d50c-7dce-4604-b9a0-c99196165b3d admin admin] RBAC: Authorizing `identity:create_grant(project_id=063c97af8ef74f7f86873dfa47c6798c, user_id=995b47fae29e4f49ba05ae3858b543c4, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:06.270542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8349d50c-7dce-4604-b9a0-c99196165b3d admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:06.341033 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1332/2658] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:13:06 2022] PUT /identity/v3/projects/063c97af8ef74f7f86873dfa47c6798c/users/995b47fae29e4f49ba05ae3858b543c4/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:06.351961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1275864b-647f-4a6c-aabf-dfb009072131 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:06.352281 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1275864b-647f-4a6c-aabf-dfb009072131 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:06.353878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1275864b-647f-4a6c-aabf-dfb009072131 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:06.395556 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-1275864b-647f-4a6c-aabf-dfb009072131 None None] MFA Rules not processed for user `995b47fae29e4f49ba05ae3858b543c4`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:06.396836 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-1275864b-647f-4a6c-aabf-dfb009072131 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:06.440644 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:06.477755 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1327/2659] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:06 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:13:06.482243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8349d50c-7dce-4604-b9a0-c99196165b3d admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:06.527241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:06.546394 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8349d50c-7dce-4604-b9a0-c99196165b3d admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:06.551162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e75cf3-790a-47c2-afea-fbd2882bbd4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:06.551787 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e75cf3-790a-47c2-afea-fbd2882bbd4a admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:06.552070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e75cf3-790a-47c2-afea-fbd2882bbd4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:06.552339 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e75cf3-790a-47c2-afea-fbd2882bbd4a admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:06.552961 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e75cf3-790a-47c2-afea-fbd2882bbd4a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:06.554877 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e75cf3-790a-47c2-afea-fbd2882bbd4a admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:06.650347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1333/2660] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:13:06 2022] POST /identity/v3/projects => generated 381 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:06.654561 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1275864b-647f-4a6c-aabf-dfb009072131 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:06.665829 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1275864b-647f-4a6c-aabf-dfb009072131 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:06.726211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762e455c-0a36-4dde-b822-7ac0a01ac102 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:06.731675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762e455c-0a36-4dde-b822-7ac0a01ac102 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:06.731675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762e455c-0a36-4dde-b822-7ac0a01ac102 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:06.731675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762e455c-0a36-4dde-b822-7ac0a01ac102 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:06.731675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762e455c-0a36-4dde-b822-7ac0a01ac102 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:06.731675 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762e455c-0a36-4dde-b822-7ac0a01ac102 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:06.818608 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1328/2661] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:13:06 2022] POST /identity/v3/users => generated 297 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:06.822638 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e75cf3-790a-47c2-afea-fbd2882bbd4a admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:06.834851 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e75cf3-790a-47c2-afea-fbd2882bbd4a admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:06.839686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0292cf02-0d00-4bb2-bf2e-5461ce6edf8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:06.840364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0292cf02-0d00-4bb2-bf2e-5461ce6edf8b admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:06.840709 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0292cf02-0d00-4bb2-bf2e-5461ce6edf8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:06.841047 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0292cf02-0d00-4bb2-bf2e-5461ce6edf8b admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:06.841814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0292cf02-0d00-4bb2-bf2e-5461ce6edf8b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:06.843476 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0292cf02-0d00-4bb2-bf2e-5461ce6edf8b admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:06.853594 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1334/2662] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:06 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:06.857948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762e455c-0a36-4dde-b822-7ac0a01ac102 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:06.868674 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762e455c-0a36-4dde-b822-7ac0a01ac102 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:06.873394 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84af8e21-19ee-49e2-85b7-18857a2bc130 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:06.874045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84af8e21-19ee-49e2-85b7-18857a2bc130 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:06.874326 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84af8e21-19ee-49e2-85b7-18857a2bc130 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:06.874600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84af8e21-19ee-49e2-85b7-18857a2bc130 admin admin] PATH_INFO: `/v3/projects/0b88568150434a3fa2a239479a5a8efa/users/85ae634fed6f48cb9597f82f61b55770/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:06.890299 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84af8e21-19ee-49e2-85b7-18857a2bc130 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b88568150434a3fa2a239479a5a8efa, user_id=85ae634fed6f48cb9597f82f61b55770, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:06.892937 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84af8e21-19ee-49e2-85b7-18857a2bc130 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:06.957499 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1329/2663] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:13:06 2022] PUT /identity/v3/projects/0b88568150434a3fa2a239479a5a8efa/users/85ae634fed6f48cb9597f82f61b55770/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:06.961525 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0292cf02-0d00-4bb2-bf2e-5461ce6edf8b admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:06.971577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0292cf02-0d00-4bb2-bf2e-5461ce6edf8b admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:07.030161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f42340-877c-48d3-926c-99d37d0b31a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:07.030687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f42340-877c-48d3-926c-99d37d0b31a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:07.030964 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f42340-877c-48d3-926c-99d37d0b31a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:07.031235 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f42340-877c-48d3-926c-99d37d0b31a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:07.031988 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f42340-877c-48d3-926c-99d37d0b31a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:07.033575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f42340-877c-48d3-926c-99d37d0b31a7 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:07.043240 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1335/2664] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:06 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:07.047633 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84af8e21-19ee-49e2-85b7-18857a2bc130 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:07.058898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84af8e21-19ee-49e2-85b7-18857a2bc130 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:07.063606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9235afd-e84c-4d1c-be66-5d0e40142988 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:07.064253 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9235afd-e84c-4d1c-be66-5d0e40142988 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:07.064535 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9235afd-e84c-4d1c-be66-5d0e40142988 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:07.064808 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9235afd-e84c-4d1c-be66-5d0e40142988 admin admin] PATH_INFO: `/v3/projects/0b88568150434a3fa2a239479a5a8efa/users/85ae634fed6f48cb9597f82f61b55770/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:07.067527 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9235afd-e84c-4d1c-be66-5d0e40142988 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b88568150434a3fa2a239479a5a8efa, user_id=85ae634fed6f48cb9597f82f61b55770, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:07.070204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9235afd-e84c-4d1c-be66-5d0e40142988 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:07.081212 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1330/2665] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:13:07 2022] PUT /identity/v3/projects/0b88568150434a3fa2a239479a5a8efa/users/85ae634fed6f48cb9597f82f61b55770/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:07.092292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08554cf8-e9ff-48a9-86ce-4e34de84e021 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:07.092584 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08554cf8-e9ff-48a9-86ce-4e34de84e021 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:07.094493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08554cf8-e9ff-48a9-86ce-4e34de84e021 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:07.121303 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-08554cf8-e9ff-48a9-86ce-4e34de84e021 None None] MFA Rules not processed for user `85ae634fed6f48cb9597f82f61b55770`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:07.122510 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-08554cf8-e9ff-48a9-86ce-4e34de84e021 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:07.189454 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1336/2666] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:07 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:07.201695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9235afd-e84c-4d1c-be66-5d0e40142988 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:07.203425 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f9235afd-e84c-4d1c-be66-5d0e40142988 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:07.262601 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9235afd-e84c-4d1c-be66-5d0e40142988 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:07.266229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236f553c-e883-4d8f-a56f-3e77ae41c084 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:07.266861 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236f553c-e883-4d8f-a56f-3e77ae41c084 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:07.267137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236f553c-e883-4d8f-a56f-3e77ae41c084 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:07.267403 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236f553c-e883-4d8f-a56f-3e77ae41c084 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:07.276482 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236f553c-e883-4d8f-a56f-3e77ae41c084 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:07.277918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236f553c-e883-4d8f-a56f-3e77ae41c084 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:07.288187 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1331/2667] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:07 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:13:07.417859 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08554cf8-e9ff-48a9-86ce-4e34de84e021 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:07.419262 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-08554cf8-e9ff-48a9-86ce-4e34de84e021 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:07.432312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08554cf8-e9ff-48a9-86ce-4e34de84e021 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:07.436144 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02085dfa-59e3-4ea6-8f24-4d0655de006f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:07.437181 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02085dfa-59e3-4ea6-8f24-4d0655de006f service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:07.437463 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02085dfa-59e3-4ea6-8f24-4d0655de006f service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:07.437737 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02085dfa-59e3-4ea6-8f24-4d0655de006f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:07.474319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:07.499909 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02085dfa-59e3-4ea6-8f24-4d0655de006f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:07.501377 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02085dfa-59e3-4ea6-8f24-4d0655de006f service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:07.510588 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1337/2668] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:07 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:13:08.071266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236f553c-e883-4d8f-a56f-3e77ae41c084 service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:08.107169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:08.135204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236f553c-e883-4d8f-a56f-3e77ae41c084 service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:08.139779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b6136b-0ebc-45db-9900-b7b96d5a7a2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:08.140411 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b6136b-0ebc-45db-9900-b7b96d5a7a2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:08.140695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b6136b-0ebc-45db-9900-b7b96d5a7a2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:08.140972 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b6136b-0ebc-45db-9900-b7b96d5a7a2a admin admin] PATH_INFO: `/v3/users/995b47fae29e4f49ba05ae3858b543c4` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:08.142141 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b6136b-0ebc-45db-9900-b7b96d5a7a2a admin admin] RBAC: Authorizing `identity:delete_user(user_id=995b47fae29e4f49ba05ae3858b543c4)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:08.144097 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b6136b-0ebc-45db-9900-b7b96d5a7a2a admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:08.388818 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-16b6136b-0ebc-45db-9900-b7b96d5a7a2a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '995b47fae29e4f49ba05ae3858b543c4'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:08.389493 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-16b6136b-0ebc-45db-9900-b7b96d5a7a2a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '995b47fae29e4f49ba05ae3858b543c4'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:08.471373 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-16b6136b-0ebc-45db-9900-b7b96d5a7a2a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '995b47fae29e4f49ba05ae3858b543c4'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:08.485028 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-16b6136b-0ebc-45db-9900-b7b96d5a7a2a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '995b47fae29e4f49ba05ae3858b543c4'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:08.485776 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-16b6136b-0ebc-45db-9900-b7b96d5a7a2a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '995b47fae29e4f49ba05ae3858b543c4'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:08.505609 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1332/2669] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:13:08 2022] DELETE /identity/v3/users/995b47fae29e4f49ba05ae3858b543c4 => generated 0 bytes in 431 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:08.507014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83614c0f-615d-425d-aaed-814145762413 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:08.507472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83614c0f-615d-425d-aaed-814145762413 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:08.507558 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83614c0f-615d-425d-aaed-814145762413 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:08.533636 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-83614c0f-615d-425d-aaed-814145762413 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:08.534848 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-83614c0f-615d-425d-aaed-814145762413 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:08.598831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1338/2670] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:13:08.608469 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b6136b-0ebc-45db-9900-b7b96d5a7a2a admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:08.609958 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-16b6136b-0ebc-45db-9900-b7b96d5a7a2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:08.668073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b6136b-0ebc-45db-9900-b7b96d5a7a2a admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:08.671690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928713ed-52f0-46c2-8e00-1b3c88223578 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:08.672290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928713ed-52f0-46c2-8e00-1b3c88223578 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:08.672570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928713ed-52f0-46c2-8e00-1b3c88223578 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:08.672841 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928713ed-52f0-46c2-8e00-1b3c88223578 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:08.682757 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928713ed-52f0-46c2-8e00-1b3c88223578 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:08.684399 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928713ed-52f0-46c2-8e00-1b3c88223578 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:08.694964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1333/2671] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:13:09.079265 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83614c0f-615d-425d-aaed-814145762413 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:09.082427 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-83614c0f-615d-425d-aaed-814145762413 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:09.093902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83614c0f-615d-425d-aaed-814145762413 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:09.099188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e62d610-29a3-4d85-805a-07956d75b0fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:09.100264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e62d610-29a3-4d85-805a-07956d75b0fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:09.100264 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e62d610-29a3-4d85-805a-07956d75b0fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:09.100475 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e62d610-29a3-4d85-805a-07956d75b0fe admin admin] PATH_INFO: `/v3/projects/063c97af8ef74f7f86873dfa47c6798c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:09.101492 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e62d610-29a3-4d85-805a-07956d75b0fe admin admin] RBAC: Authorizing `identity:delete_project(project_id=063c97af8ef74f7f86873dfa47c6798c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:09.103416 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e62d610-29a3-4d85-805a-07956d75b0fe admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:09.192396 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7e62d610-29a3-4d85-805a-07956d75b0fe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '063c97af8ef74f7f86873dfa47c6798c', 'request_id': 'req-7e62d610-29a3-4d85-805a-07956d75b0fe'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:09.198146 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7e62d610-29a3-4d85-805a-07956d75b0fe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '063c97af8ef74f7f86873dfa47c6798c', 'request_id': 'req-7e62d610-29a3-4d85-805a-07956d75b0fe'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:09.204385 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7e62d610-29a3-4d85-805a-07956d75b0fe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '063c97af8ef74f7f86873dfa47c6798c', 'request_id': 'req-7e62d610-29a3-4d85-805a-07956d75b0fe'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:09.276172 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7e62d610-29a3-4d85-805a-07956d75b0fe admin admin] The token cache is being invalidate because project 063c97af8ef74f7f86873dfa47c6798c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:13:09.276457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7e62d610-29a3-4d85-805a-07956d75b0fe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:09.277103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-7e62d610-29a3-4d85-805a-07956d75b0fe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:09.278637 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1339/2672] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:13:09 2022] DELETE /identity/v3/projects/063c97af8ef74f7f86873dfa47c6798c => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:09.282738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928713ed-52f0-46c2-8e00-1b3c88223578 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:09.293065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928713ed-52f0-46c2-8e00-1b3c88223578 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:09.341483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:09.348549 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149f58a2-7187-477b-b258-b113a7fd08a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:09.349202 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149f58a2-7187-477b-b258-b113a7fd08a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:09.349482 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149f58a2-7187-477b-b258-b113a7fd08a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:09.349750 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149f58a2-7187-477b-b258-b113a7fd08a7 admin admin] PATH_INFO: `/v3/users/85ae634fed6f48cb9597f82f61b55770` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:09.350877 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149f58a2-7187-477b-b258-b113a7fd08a7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=85ae634fed6f48cb9597f82f61b55770)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:09.352784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149f58a2-7187-477b-b258-b113a7fd08a7 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:09.490860 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-149f58a2-7187-477b-b258-b113a7fd08a7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '85ae634fed6f48cb9597f82f61b55770'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:09.491518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-149f58a2-7187-477b-b258-b113a7fd08a7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '85ae634fed6f48cb9597f82f61b55770'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:09.557205 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-149f58a2-7187-477b-b258-b113a7fd08a7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '85ae634fed6f48cb9597f82f61b55770'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:09.570441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-149f58a2-7187-477b-b258-b113a7fd08a7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '85ae634fed6f48cb9597f82f61b55770'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:09.571098 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-149f58a2-7187-477b-b258-b113a7fd08a7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '85ae634fed6f48cb9597f82f61b55770'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:09.583698 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1334/2673] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:13:09 2022] DELETE /identity/v3/users/85ae634fed6f48cb9597f82f61b55770 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:09.974949 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e62d610-29a3-4d85-805a-07956d75b0fe admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:09.976385 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7e62d610-29a3-4d85-805a-07956d75b0fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:09.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e62d610-29a3-4d85-805a-07956d75b0fe admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:10.045054 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:10.051316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8960231-841c-4422-acbf-6472b2ad29a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:10.052103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8960231-841c-4422-acbf-6472b2ad29a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:10.052383 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8960231-841c-4422-acbf-6472b2ad29a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:10.057577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8960231-841c-4422-acbf-6472b2ad29a1 admin admin] PATH_INFO: `/v3/projects/0b88568150434a3fa2a239479a5a8efa` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:10.057577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8960231-841c-4422-acbf-6472b2ad29a1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b88568150434a3fa2a239479a5a8efa)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:10.057577 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8960231-841c-4422-acbf-6472b2ad29a1 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:10.151007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e8960231-841c-4422-acbf-6472b2ad29a1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b88568150434a3fa2a239479a5a8efa', 'request_id': 'req-e8960231-841c-4422-acbf-6472b2ad29a1'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:10.156289 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e8960231-841c-4422-acbf-6472b2ad29a1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b88568150434a3fa2a239479a5a8efa', 'request_id': 'req-e8960231-841c-4422-acbf-6472b2ad29a1'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:10.162641 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e8960231-841c-4422-acbf-6472b2ad29a1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b88568150434a3fa2a239479a5a8efa', 'request_id': 'req-e8960231-841c-4422-acbf-6472b2ad29a1'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:10.259510 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e8960231-841c-4422-acbf-6472b2ad29a1 admin admin] The token cache is being invalidate because project 0b88568150434a3fa2a239479a5a8efa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:13:10.259704 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e8960231-841c-4422-acbf-6472b2ad29a1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:10.260332 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e8960231-841c-4422-acbf-6472b2ad29a1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:10.261852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1340/2674] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:13:09 2022] DELETE /identity/v3/projects/0b88568150434a3fa2a239479a5a8efa => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:10.266721 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30cd051-d57e-44f5-b083-d7eb39367a98 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:10.267042 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30cd051-d57e-44f5-b083-d7eb39367a98 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:10.267325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30cd051-d57e-44f5-b083-d7eb39367a98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:10.287099 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-b30cd051-d57e-44f5-b083-d7eb39367a98 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:10.288315 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b30cd051-d57e-44f5-b083-d7eb39367a98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:10.347297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1335/2675] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:10.356057 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b1a3f7-9201-4e3c-8f10-9da9608f391f None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:10.356413 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b1a3f7-9201-4e3c-8f10-9da9608f391f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:10.356804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b1a3f7-9201-4e3c-8f10-9da9608f391f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:10.382303 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-09b1a3f7-9201-4e3c-8f10-9da9608f391f None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:10.383623 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-09b1a3f7-9201-4e3c-8f10-9da9608f391f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:10.391370 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1341/2676] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:10.396279 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30cd051-d57e-44f5-b083-d7eb39367a98 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:10.405880 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30cd051-d57e-44f5-b083-d7eb39367a98 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:10.410438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf91660-a423-4096-8173-8e32ef3acaaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:10.411027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf91660-a423-4096-8173-8e32ef3acaaa admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:10.411594 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf91660-a423-4096-8173-8e32ef3acaaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:10.411870 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf91660-a423-4096-8173-8e32ef3acaaa admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:10.412522 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf91660-a423-4096-8173-8e32ef3acaaa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:13:10.412824 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf91660-a423-4096-8173-8e32ef3acaaa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:10.414456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf91660-a423-4096-8173-8e32ef3acaaa admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:10.425815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1336/2677] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:13:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:13:10.430137 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b1a3f7-9201-4e3c-8f10-9da9608f391f None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:10.439708 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b1a3f7-9201-4e3c-8f10-9da9608f391f None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:10.444947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b67555-7344-4314-944e-a26662e60613 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:10.445582 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b67555-7344-4314-944e-a26662e60613 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:10.445865 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b67555-7344-4314-944e-a26662e60613 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:10.446139 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b67555-7344-4314-944e-a26662e60613 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:10.446774 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b67555-7344-4314-944e-a26662e60613 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:10.448674 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b67555-7344-4314-944e-a26662e60613 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:10.568135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1342/2678] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:13:10 2022] POST /identity/v3/projects => generated 377 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:13:10.568596 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf91660-a423-4096-8173-8e32ef3acaaa admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:10.578849 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf91660-a423-4096-8173-8e32ef3acaaa admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:10.635371 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7bd2d6-e880-4947-a29c-8cff60e2856e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:10.635980 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7bd2d6-e880-4947-a29c-8cff60e2856e admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:10.636290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7bd2d6-e880-4947-a29c-8cff60e2856e admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:10.636565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7bd2d6-e880-4947-a29c-8cff60e2856e admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:10.637235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7bd2d6-e880-4947-a29c-8cff60e2856e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:10.639144 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7bd2d6-e880-4947-a29c-8cff60e2856e admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:10.648729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:10.712939 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1337/2679] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:13:10 2022] POST /identity/v3/users => generated 289 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:10.717121 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b67555-7344-4314-944e-a26662e60613 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:10.728367 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b67555-7344-4314-944e-a26662e60613 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:10.733194 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f1801a-e8f6-4219-848b-f0b2e7b70e8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:10.733788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f1801a-e8f6-4219-848b-f0b2e7b70e8c admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:10.734070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f1801a-e8f6-4219-848b-f0b2e7b70e8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:10.734344 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f1801a-e8f6-4219-848b-f0b2e7b70e8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:10.735022 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f1801a-e8f6-4219-848b-f0b2e7b70e8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:10.736623 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f1801a-e8f6-4219-848b-f0b2e7b70e8c admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:10.747001 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1343/2680] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:10.750977 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7bd2d6-e880-4947-a29c-8cff60e2856e admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:10.761432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7bd2d6-e880-4947-a29c-8cff60e2856e admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:10.765963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7cd1fe-bcaf-4542-b15c-0d95961c8c59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:10.766543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7cd1fe-bcaf-4542-b15c-0d95961c8c59 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:10.766821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7cd1fe-bcaf-4542-b15c-0d95961c8c59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:10.767097 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7cd1fe-bcaf-4542-b15c-0d95961c8c59 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:10.767739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7cd1fe-bcaf-4542-b15c-0d95961c8c59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:10.769890 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7cd1fe-bcaf-4542-b15c-0d95961c8c59 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:10.780175 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1338/2681] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:10.784035 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f1801a-e8f6-4219-848b-f0b2e7b70e8c admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:10.794522 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f1801a-e8f6-4219-848b-f0b2e7b70e8c admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:10.799403 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128f37eb-8f1f-4645-8857-145d30edc083 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:10.800018 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128f37eb-8f1f-4645-8857-145d30edc083 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:10.800297 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128f37eb-8f1f-4645-8857-145d30edc083 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:10.800572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128f37eb-8f1f-4645-8857-145d30edc083 admin admin] PATH_INFO: `/v3/projects/a5edd00a8bd348cca2077c065556d42b/users/aaba6d8a13eb4def8755cd5267d57a31/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:10.815472 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128f37eb-8f1f-4645-8857-145d30edc083 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5edd00a8bd348cca2077c065556d42b, user_id=aaba6d8a13eb4def8755cd5267d57a31, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:10.818332 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128f37eb-8f1f-4645-8857-145d30edc083 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:10.888298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1344/2682] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:13:10 2022] PUT /identity/v3/projects/a5edd00a8bd348cca2077c065556d42b/users/aaba6d8a13eb4def8755cd5267d57a31/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:10.899308 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cad06ad-4371-4cd2-b029-07f7ba6764da None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:10.899594 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cad06ad-4371-4cd2-b029-07f7ba6764da None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:10.899871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cad06ad-4371-4cd2-b029-07f7ba6764da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:10.928081 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-8cad06ad-4371-4cd2-b029-07f7ba6764da None None] MFA Rules not processed for user `aaba6d8a13eb4def8755cd5267d57a31`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:10.929335 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8cad06ad-4371-4cd2-b029-07f7ba6764da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:10.993031 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1339/2683] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:10 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:11.004546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128f37eb-8f1f-4645-8857-145d30edc083 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:11.006431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-128f37eb-8f1f-4645-8857-145d30edc083 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:11.063151 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128f37eb-8f1f-4645-8857-145d30edc083 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:11.067328 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5128c1-aa4e-48b3-83a7-d4e515a758fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:11.067933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5128c1-aa4e-48b3-83a7-d4e515a758fd service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:11.068211 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5128c1-aa4e-48b3-83a7-d4e515a758fd service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:11.068489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5128c1-aa4e-48b3-83a7-d4e515a758fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:11.076603 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5128c1-aa4e-48b3-83a7-d4e515a758fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:11.078071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5128c1-aa4e-48b3-83a7-d4e515a758fd service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:11.087375 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1345/2684] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:11 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:13:13.322779 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cad06ad-4371-4cd2-b029-07f7ba6764da None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:13.324344 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8cad06ad-4371-4cd2-b029-07f7ba6764da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:13.398257 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cad06ad-4371-4cd2-b029-07f7ba6764da None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:13.402912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31d32ac-f2fd-4ff2-997d-5b9eee38ac5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:13.403626 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31d32ac-f2fd-4ff2-997d-5b9eee38ac5d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:13.404099 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31d32ac-f2fd-4ff2-997d-5b9eee38ac5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:13.404232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31d32ac-f2fd-4ff2-997d-5b9eee38ac5d admin admin] PATH_INFO: `/v3/users/c106b19876d5474e94281cd06ee7316a` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:13.405499 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31d32ac-f2fd-4ff2-997d-5b9eee38ac5d admin admin] RBAC: Authorizing `identity:delete_user(user_id=c106b19876d5474e94281cd06ee7316a)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:13.407424 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31d32ac-f2fd-4ff2-997d-5b9eee38ac5d admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:13.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d31d32ac-f2fd-4ff2-997d-5b9eee38ac5d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c106b19876d5474e94281cd06ee7316a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:13.567370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d31d32ac-f2fd-4ff2-997d-5b9eee38ac5d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c106b19876d5474e94281cd06ee7316a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:13.620219 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d31d32ac-f2fd-4ff2-997d-5b9eee38ac5d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c106b19876d5474e94281cd06ee7316a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:13.634111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d31d32ac-f2fd-4ff2-997d-5b9eee38ac5d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c106b19876d5474e94281cd06ee7316a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:13.634911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-d31d32ac-f2fd-4ff2-997d-5b9eee38ac5d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c106b19876d5474e94281cd06ee7316a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:13.636155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:INSERT=3,keystone:SELECT=87,keystone:DELETE=10 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:13.648168 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1340/2685] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:13:13 2022] DELETE /identity/v3/users/c106b19876d5474e94281cd06ee7316a => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:13.652608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b99ea4-40cc-423d-8b5d-043b8623e6ca None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:13.653128 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b99ea4-40cc-423d-8b5d-043b8623e6ca None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:13.653223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b99ea4-40cc-423d-8b5d-043b8623e6ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:13.673110 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-d0b99ea4-40cc-423d-8b5d-043b8623e6ca None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:13.674381 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d0b99ea4-40cc-423d-8b5d-043b8623e6ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:13.695155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:13.736957 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1346/2686] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:13.747605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31d32ac-f2fd-4ff2-997d-5b9eee38ac5d admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:13.749110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d31d32ac-f2fd-4ff2-997d-5b9eee38ac5d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:13.820034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31d32ac-f2fd-4ff2-997d-5b9eee38ac5d admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:13.827149 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bb7864-2596-493b-a036-36438058a2c6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:13.827767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bb7864-2596-493b-a036-36438058a2c6 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:13.828052 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bb7864-2596-493b-a036-36438058a2c6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:13.828359 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bb7864-2596-493b-a036-36438058a2c6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:13.838592 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bb7864-2596-493b-a036-36438058a2c6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:13.840021 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bb7864-2596-493b-a036-36438058a2c6 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:13.850114 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1341/2687] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:13:14.256343 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b99ea4-40cc-423d-8b5d-043b8623e6ca None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:14.257903 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d0b99ea4-40cc-423d-8b5d-043b8623e6ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:14.276050 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b99ea4-40cc-423d-8b5d-043b8623e6ca None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:14.283983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:14.283983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:14.283983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:14.283983 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58 admin admin] PATH_INFO: `/v3/projects/6841cb8436ae43ff843cf938e91711ab` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:14.284793 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6841cb8436ae43ff843cf938e91711ab)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:14.285217 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:14.375254 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6841cb8436ae43ff843cf938e91711ab', 'request_id': 'req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:14.381416 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6841cb8436ae43ff843cf938e91711ab', 'request_id': 'req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:14.387721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6841cb8436ae43ff843cf938e91711ab', 'request_id': 'req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:14.448684 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58 admin admin] The token cache is being invalidate because project 6841cb8436ae43ff843cf938e91711ab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:13:14.448984 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:14.449626 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:14.451100 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1347/2688] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:13:14 2022] DELETE /identity/v3/projects/6841cb8436ae43ff843cf938e91711ab => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:14.455084 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bb7864-2596-493b-a036-36438058a2c6 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:14.465485 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bb7864-2596-493b-a036-36438058a2c6 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:14.522282 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efd8eeb-8add-4ce2-bb5a-2be9f10ce1fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:14.522903 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efd8eeb-8add-4ce2-bb5a-2be9f10ce1fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:14.523184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efd8eeb-8add-4ce2-bb5a-2be9f10ce1fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:14.523454 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efd8eeb-8add-4ce2-bb5a-2be9f10ce1fa admin admin] PATH_INFO: `/v3/users/51a68bcdc8974e9c9ded50b0c0975157` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:14.524546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efd8eeb-8add-4ce2-bb5a-2be9f10ce1fa admin admin] RBAC: Authorizing `identity:delete_user(user_id=51a68bcdc8974e9c9ded50b0c0975157)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:14.526523 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efd8eeb-8add-4ce2-bb5a-2be9f10ce1fa admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:14.696669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7efd8eeb-8add-4ce2-bb5a-2be9f10ce1fa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '51a68bcdc8974e9c9ded50b0c0975157'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:14.697276 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7efd8eeb-8add-4ce2-bb5a-2be9f10ce1fa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '51a68bcdc8974e9c9ded50b0c0975157'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:14.774336 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7efd8eeb-8add-4ce2-bb5a-2be9f10ce1fa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '51a68bcdc8974e9c9ded50b0c0975157'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:14.788432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7efd8eeb-8add-4ce2-bb5a-2be9f10ce1fa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '51a68bcdc8974e9c9ded50b0c0975157'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:14.789165 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7efd8eeb-8add-4ce2-bb5a-2be9f10ce1fa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '51a68bcdc8974e9c9ded50b0c0975157'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:14.802708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1342/2689] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:13:14 2022] DELETE /identity/v3/users/51a68bcdc8974e9c9ded50b0c0975157 => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:15.268335 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:15.268818 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:15.282423 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8ca3d6-8756-413e-b4c3-96c7b043cb58 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:15.341463 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa649361-b65f-4167-bfb3-24a96f4f61b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:15.342093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa649361-b65f-4167-bfb3-24a96f4f61b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:15.342376 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa649361-b65f-4167-bfb3-24a96f4f61b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:15.342656 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa649361-b65f-4167-bfb3-24a96f4f61b1 admin admin] PATH_INFO: `/v3/projects/7ca56a8fae1747c6b2a17b632cb8d430` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:15.343779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa649361-b65f-4167-bfb3-24a96f4f61b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7ca56a8fae1747c6b2a17b632cb8d430)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:15.345739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa649361-b65f-4167-bfb3-24a96f4f61b1 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:15.489659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-fa649361-b65f-4167-bfb3-24a96f4f61b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7ca56a8fae1747c6b2a17b632cb8d430', 'request_id': 'req-fa649361-b65f-4167-bfb3-24a96f4f61b1'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:15.491555 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:15.495720 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-fa649361-b65f-4167-bfb3-24a96f4f61b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7ca56a8fae1747c6b2a17b632cb8d430', 'request_id': 'req-fa649361-b65f-4167-bfb3-24a96f4f61b1'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:15.501749 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-fa649361-b65f-4167-bfb3-24a96f4f61b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7ca56a8fae1747c6b2a17b632cb8d430', 'request_id': 'req-fa649361-b65f-4167-bfb3-24a96f4f61b1'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:15.596154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-fa649361-b65f-4167-bfb3-24a96f4f61b1 admin admin] The token cache is being invalidate because project 7ca56a8fae1747c6b2a17b632cb8d430 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:13:15.596437 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-fa649361-b65f-4167-bfb3-24a96f4f61b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:15.597048 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-fa649361-b65f-4167-bfb3-24a96f4f61b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:15.599291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1348/2690] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:13:15 2022] DELETE /identity/v3/projects/7ca56a8fae1747c6b2a17b632cb8d430 => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:15.603351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efd8eeb-8add-4ce2-bb5a-2be9f10ce1fa admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:15.604844 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-7efd8eeb-8add-4ce2-bb5a-2be9f10ce1fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:15.618709 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efd8eeb-8add-4ce2-bb5a-2be9f10ce1fa admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:15.626554 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:15.675088 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a6031e-bb95-43c7-980b-fc616f2fe8d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:15.676500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a6031e-bb95-43c7-980b-fc616f2fe8d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:15.676773 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a6031e-bb95-43c7-980b-fc616f2fe8d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:15.677053 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a6031e-bb95-43c7-980b-fc616f2fe8d7 admin admin] PATH_INFO: `/v3/users/3c8ab6c7e08349b699c7ee9366b1c969` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:15.678208 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a6031e-bb95-43c7-980b-fc616f2fe8d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3c8ab6c7e08349b699c7ee9366b1c969)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:15.680140 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a6031e-bb95-43c7-980b-fc616f2fe8d7 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:15.815371 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-86a6031e-bb95-43c7-980b-fc616f2fe8d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3c8ab6c7e08349b699c7ee9366b1c969'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:15.816131 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-86a6031e-bb95-43c7-980b-fc616f2fe8d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3c8ab6c7e08349b699c7ee9366b1c969'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:15.882647 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-86a6031e-bb95-43c7-980b-fc616f2fe8d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3c8ab6c7e08349b699c7ee9366b1c969'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:15.895180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-86a6031e-bb95-43c7-980b-fc616f2fe8d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3c8ab6c7e08349b699c7ee9366b1c969'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:15.895837 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-86a6031e-bb95-43c7-980b-fc616f2fe8d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3c8ab6c7e08349b699c7ee9366b1c969'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:15.908015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1343/2691] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:13:15 2022] DELETE /identity/v3/users/3c8ab6c7e08349b699c7ee9366b1c969 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:16.267920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa649361-b65f-4167-bfb3-24a96f4f61b1 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:16.270595 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-fa649361-b65f-4167-bfb3-24a96f4f61b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:16.299224 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa649361-b65f-4167-bfb3-24a96f4f61b1 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:16.324278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a6031e-bb95-43c7-980b-fc616f2fe8d7 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:16.325698 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-86a6031e-bb95-43c7-980b-fc616f2fe8d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:16.362174 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f823a2f5-271d-4f11-8288-5ad0565804aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:16.362878 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f823a2f5-271d-4f11-8288-5ad0565804aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:16.363661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f823a2f5-271d-4f11-8288-5ad0565804aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:16.363936 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f823a2f5-271d-4f11-8288-5ad0565804aa admin admin] PATH_INFO: `/v3/projects/136206f1ccd544a5b27352822a9de9fc` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:16.365061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f823a2f5-271d-4f11-8288-5ad0565804aa admin admin] RBAC: Authorizing `identity:delete_project(project_id=136206f1ccd544a5b27352822a9de9fc)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:16.366998 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f823a2f5-271d-4f11-8288-5ad0565804aa admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:16.428594 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a6031e-bb95-43c7-980b-fc616f2fe8d7 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:16.433767 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbde0ef-f091-4512-bb40-47759ef6ac3d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b29f444d56ec49308f7fef16f4d1eea5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:16.434380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbde0ef-f091-4512-bb40-47759ef6ac3d service placement] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:16.434663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbde0ef-f091-4512-bb40-47759ef6ac3d service placement] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:16.434938 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbde0ef-f091-4512-bb40-47759ef6ac3d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:16.436695 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6fbde0ef-f091-4512-bb40-47759ef6ac3d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:16.448362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbde0ef-f091-4512-bb40-47759ef6ac3d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:16.449959 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbde0ef-f091-4512-bb40-47759ef6ac3d service placement] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:16.455257 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:16.460180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1344/2692] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:16 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:13:16.471291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f823a2f5-271d-4f11-8288-5ad0565804aa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '136206f1ccd544a5b27352822a9de9fc', 'request_id': 'req-f823a2f5-271d-4f11-8288-5ad0565804aa'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:16.477470 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f823a2f5-271d-4f11-8288-5ad0565804aa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '136206f1ccd544a5b27352822a9de9fc', 'request_id': 'req-f823a2f5-271d-4f11-8288-5ad0565804aa'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:16.483697 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f823a2f5-271d-4f11-8288-5ad0565804aa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '136206f1ccd544a5b27352822a9de9fc', 'request_id': 'req-f823a2f5-271d-4f11-8288-5ad0565804aa'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:16.573226 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f823a2f5-271d-4f11-8288-5ad0565804aa admin admin] The token cache is being invalidate because project 136206f1ccd544a5b27352822a9de9fc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:13:16.573509 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f823a2f5-271d-4f11-8288-5ad0565804aa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:16.574180 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f823a2f5-271d-4f11-8288-5ad0565804aa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:16.575602 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1349/2693] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:13:16 2022] DELETE /identity/v3/projects/136206f1ccd544a5b27352822a9de9fc => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:16.581502 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5504d664-f926-444a-b82d-e3435d45685a None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:16.582491 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5504d664-f926-444a-b82d-e3435d45685a None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:16.582777 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5504d664-f926-444a-b82d-e3435d45685a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:16.603677 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-5504d664-f926-444a-b82d-e3435d45685a None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:16.604911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-5504d664-f926-444a-b82d-e3435d45685a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:16.667672 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1345/2694] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:16.676804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f62cc82-031b-47bf-8635-3cb288a28e27 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:16.677104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f62cc82-031b-47bf-8635-3cb288a28e27 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:16.677388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f62cc82-031b-47bf-8635-3cb288a28e27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:16.698347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-1f62cc82-031b-47bf-8635-3cb288a28e27 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:16.699682 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-1f62cc82-031b-47bf-8635-3cb288a28e27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:16.708346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1350/2695] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:16.713165 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5504d664-f926-444a-b82d-e3435d45685a None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:16.723388 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5504d664-f926-444a-b82d-e3435d45685a None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:16.730534 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9d146e-2a84-4e9d-8d1f-8b3f3586dea5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:16.732189 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9d146e-2a84-4e9d-8d1f-8b3f3586dea5 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:16.732189 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9d146e-2a84-4e9d-8d1f-8b3f3586dea5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:16.732387 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9d146e-2a84-4e9d-8d1f-8b3f3586dea5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:16.732912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9d146e-2a84-4e9d-8d1f-8b3f3586dea5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:13:16.733258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9d146e-2a84-4e9d-8d1f-8b3f3586dea5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:16.734895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9d146e-2a84-4e9d-8d1f-8b3f3586dea5 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:16.748401 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1346/2696] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:13:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:13:16.753029 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f62cc82-031b-47bf-8635-3cb288a28e27 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:16.763392 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f62cc82-031b-47bf-8635-3cb288a28e27 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:16.768219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d12f58-b032-4b1c-b36c-bb8702c2e2ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:16.768819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d12f58-b032-4b1c-b36c-bb8702c2e2ed admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:16.769329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d12f58-b032-4b1c-b36c-bb8702c2e2ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:16.769523 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d12f58-b032-4b1c-b36c-bb8702c2e2ed admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:16.770402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d12f58-b032-4b1c-b36c-bb8702c2e2ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:16.772349 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d12f58-b032-4b1c-b36c-bb8702c2e2ed admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:16.873658 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1351/2697] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:13:16 2022] POST /identity/v3/projects => generated 369 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:16.877837 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9d146e-2a84-4e9d-8d1f-8b3f3586dea5 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:16.888509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9d146e-2a84-4e9d-8d1f-8b3f3586dea5 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:16.945645 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ea335d-090e-4fc0-9cd4-79dc6a221746 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:16.946169 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ea335d-090e-4fc0-9cd4-79dc6a221746 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:16.946451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ea335d-090e-4fc0-9cd4-79dc6a221746 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:16.946733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ea335d-090e-4fc0-9cd4-79dc6a221746 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:16.947352 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ea335d-090e-4fc0-9cd4-79dc6a221746 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:16.949280 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ea335d-090e-4fc0-9cd4-79dc6a221746 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:17.024088 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1347/2698] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:13:16 2022] POST /identity/v3/users => generated 285 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:17.027579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d12f58-b032-4b1c-b36c-bb8702c2e2ed admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:17.039207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d12f58-b032-4b1c-b36c-bb8702c2e2ed admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:17.044025 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5757a704-bfe4-4e3a-87eb-835c8a32a813 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:17.044624 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5757a704-bfe4-4e3a-87eb-835c8a32a813 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:17.044903 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5757a704-bfe4-4e3a-87eb-835c8a32a813 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:17.045215 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5757a704-bfe4-4e3a-87eb-835c8a32a813 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:17.045861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5757a704-bfe4-4e3a-87eb-835c8a32a813 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:17.047452 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5757a704-bfe4-4e3a-87eb-835c8a32a813 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:17.062861 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1352/2699] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:17 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:17.064204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ea335d-090e-4fc0-9cd4-79dc6a221746 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:17.075188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ea335d-090e-4fc0-9cd4-79dc6a221746 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:17.079828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7806cf10-442c-4db9-8c48-cd4698e35bbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:17.080362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7806cf10-442c-4db9-8c48-cd4698e35bbc admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:17.080646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7806cf10-442c-4db9-8c48-cd4698e35bbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:17.080948 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7806cf10-442c-4db9-8c48-cd4698e35bbc admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:17.081707 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7806cf10-442c-4db9-8c48-cd4698e35bbc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:17.083987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7806cf10-442c-4db9-8c48-cd4698e35bbc admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:17.099321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1348/2700] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:17 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:17.100715 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5757a704-bfe4-4e3a-87eb-835c8a32a813 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:17.111666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5757a704-bfe4-4e3a-87eb-835c8a32a813 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:17.116928 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313f8e7b-2f07-49ae-bbb4-f01c9961ea84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:17.117930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313f8e7b-2f07-49ae-bbb4-f01c9961ea84 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:17.118249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313f8e7b-2f07-49ae-bbb4-f01c9961ea84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:17.118526 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313f8e7b-2f07-49ae-bbb4-f01c9961ea84 admin admin] PATH_INFO: `/v3/projects/9316dc6b766147b4ac43791e2714f2e5/users/4dbfca17abc04d4580b2997896a94776/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:17.136188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313f8e7b-2f07-49ae-bbb4-f01c9961ea84 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9316dc6b766147b4ac43791e2714f2e5, user_id=4dbfca17abc04d4580b2997896a94776, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:17.139062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313f8e7b-2f07-49ae-bbb4-f01c9961ea84 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:17.199075 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1353/2701] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:13:17 2022] PUT /identity/v3/projects/9316dc6b766147b4ac43791e2714f2e5/users/4dbfca17abc04d4580b2997896a94776/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:17.216031 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee939d58-5cf9-4a0d-a6b7-0690dfed86c1 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:17.216570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee939d58-5cf9-4a0d-a6b7-0690dfed86c1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:17.216920 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee939d58-5cf9-4a0d-a6b7-0690dfed86c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:17.248266 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-ee939d58-5cf9-4a0d-a6b7-0690dfed86c1 None None] MFA Rules not processed for user `4dbfca17abc04d4580b2997896a94776`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:17.249571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ee939d58-5cf9-4a0d-a6b7-0690dfed86c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:17.306504 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:17.318376 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1349/2702] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:17 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:17.329181 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313f8e7b-2f07-49ae-bbb4-f01c9961ea84 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:17.330874 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-313f8e7b-2f07-49ae-bbb4-f01c9961ea84 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:17.342289 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:17.390845 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313f8e7b-2f07-49ae-bbb4-f01c9961ea84 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:17.396049 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06076622-640e-42ae-9ac3-db0e1f3ebfec service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '69153344fc4b45588bdbcecdfcfd7c3f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:17.396768 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06076622-640e-42ae-9ac3-db0e1f3ebfec service glance] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:17.397158 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06076622-640e-42ae-9ac3-db0e1f3ebfec service glance] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:17.397438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06076622-640e-42ae-9ac3-db0e1f3ebfec service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:17.407552 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06076622-640e-42ae-9ac3-db0e1f3ebfec service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:17.409355 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06076622-640e-42ae-9ac3-db0e1f3ebfec service glance] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:17.423455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1354/2703] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:17 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:13:27.362826 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:27.446203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=23 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:35.259284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee939d58-5cf9-4a0d-a6b7-0690dfed86c1 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:35.260698 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-ee939d58-5cf9-4a0d-a6b7-0690dfed86c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:35.341567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee939d58-5cf9-4a0d-a6b7-0690dfed86c1 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:35.341567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b215d608-66bf-467e-a6a1-45ed60b2d027 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:35.341567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b215d608-66bf-467e-a6a1-45ed60b2d027 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:35.341567 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b215d608-66bf-467e-a6a1-45ed60b2d027 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:35.341993 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b215d608-66bf-467e-a6a1-45ed60b2d027 admin admin] PATH_INFO: `/v3/users/aaba6d8a13eb4def8755cd5267d57a31` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:35.341993 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b215d608-66bf-467e-a6a1-45ed60b2d027 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aaba6d8a13eb4def8755cd5267d57a31)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:35.341993 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b215d608-66bf-467e-a6a1-45ed60b2d027 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:35.578268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b215d608-66bf-467e-a6a1-45ed60b2d027 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aaba6d8a13eb4def8755cd5267d57a31'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:35.579198 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b215d608-66bf-467e-a6a1-45ed60b2d027 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aaba6d8a13eb4def8755cd5267d57a31'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:35.631525 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b215d608-66bf-467e-a6a1-45ed60b2d027 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aaba6d8a13eb4def8755cd5267d57a31'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:35.645106 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b215d608-66bf-467e-a6a1-45ed60b2d027 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aaba6d8a13eb4def8755cd5267d57a31'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:35.645775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-b215d608-66bf-467e-a6a1-45ed60b2d027 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aaba6d8a13eb4def8755cd5267d57a31'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:35.660179 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1350/2704] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:13:35 2022] DELETE /identity/v3/users/aaba6d8a13eb4def8755cd5267d57a31 => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:35.664679 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f90bcaf-0f15-464f-9989-15f205756d7b None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:35.666222 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f90bcaf-0f15-464f-9989-15f205756d7b None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:35.666222 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f90bcaf-0f15-464f-9989-15f205756d7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:35.684847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-0f90bcaf-0f15-464f-9989-15f205756d7b None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:35.686118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0f90bcaf-0f15-464f-9989-15f205756d7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:35.751259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1355/2705] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:35.761460 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b215d608-66bf-467e-a6a1-45ed60b2d027 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:35.762909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b215d608-66bf-467e-a6a1-45ed60b2d027 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:35.841003 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b215d608-66bf-467e-a6a1-45ed60b2d027 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:35.845909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac68aef1-64a1-4889-97c1-08f5b9bf5c7c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:35.846706 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac68aef1-64a1-4889-97c1-08f5b9bf5c7c service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:35.846989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac68aef1-64a1-4889-97c1-08f5b9bf5c7c service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:35.847259 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac68aef1-64a1-4889-97c1-08f5b9bf5c7c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:35.856546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac68aef1-64a1-4889-97c1-08f5b9bf5c7c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:35.858000 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac68aef1-64a1-4889-97c1-08f5b9bf5c7c service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:35.868213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1351/2706] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:13:36.262177 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f90bcaf-0f15-464f-9989-15f205756d7b None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:36.263591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0f90bcaf-0f15-464f-9989-15f205756d7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:36.277301 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f90bcaf-0f15-464f-9989-15f205756d7b None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:36.287535 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe133dde-c270-4a7a-a52e-1909cdee0027 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:36.288210 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe133dde-c270-4a7a-a52e-1909cdee0027 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:36.288624 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe133dde-c270-4a7a-a52e-1909cdee0027 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:36.299622 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe133dde-c270-4a7a-a52e-1909cdee0027 admin admin] PATH_INFO: `/v3/projects/a5edd00a8bd348cca2077c065556d42b` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:36.307214 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe133dde-c270-4a7a-a52e-1909cdee0027 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a5edd00a8bd348cca2077c065556d42b)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:36.309179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe133dde-c270-4a7a-a52e-1909cdee0027 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:36.415053 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-fe133dde-c270-4a7a-a52e-1909cdee0027 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a5edd00a8bd348cca2077c065556d42b', 'request_id': 'req-fe133dde-c270-4a7a-a52e-1909cdee0027'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:36.421268 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-fe133dde-c270-4a7a-a52e-1909cdee0027 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a5edd00a8bd348cca2077c065556d42b', 'request_id': 'req-fe133dde-c270-4a7a-a52e-1909cdee0027'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:36.427578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-fe133dde-c270-4a7a-a52e-1909cdee0027 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a5edd00a8bd348cca2077c065556d42b', 'request_id': 'req-fe133dde-c270-4a7a-a52e-1909cdee0027'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:36.475993 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-fe133dde-c270-4a7a-a52e-1909cdee0027 admin admin] The token cache is being invalidate because project a5edd00a8bd348cca2077c065556d42b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:13:36.476341 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-fe133dde-c270-4a7a-a52e-1909cdee0027 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:36.476963 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-fe133dde-c270-4a7a-a52e-1909cdee0027 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:36.478466 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1356/2707] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:13:36 2022] DELETE /identity/v3/projects/a5edd00a8bd348cca2077c065556d42b => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:36.483988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957f5b86-da0d-4430-b955-43abe3773ac7 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:36.484454 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957f5b86-da0d-4430-b955-43abe3773ac7 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:36.484569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957f5b86-da0d-4430-b955-43abe3773ac7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:36.492267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:36.505443 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-957f5b86-da0d-4430-b955-43abe3773ac7 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:36.506589 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-957f5b86-da0d-4430-b955-43abe3773ac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:36.566907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1352/2708] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:13:36.576375 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5a3bd3-fc3c-4b42-bb06-77e1e3aa450c None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:36.576654 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5a3bd3-fc3c-4b42-bb06-77e1e3aa450c None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:36.576923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5a3bd3-fc3c-4b42-bb06-77e1e3aa450c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:36.596779 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-aa5a3bd3-fc3c-4b42-bb06-77e1e3aa450c None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:36.598010 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-aa5a3bd3-fc3c-4b42-bb06-77e1e3aa450c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:36.605802 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1357/2709] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:13:36.610041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957f5b86-da0d-4430-b955-43abe3773ac7 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:36.619759 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957f5b86-da0d-4430-b955-43abe3773ac7 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:36.624576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c922ec-52a3-4b81-b358-62a41b874da6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:36.625189 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c922ec-52a3-4b81-b358-62a41b874da6 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:36.625484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c922ec-52a3-4b81-b358-62a41b874da6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:36.625737 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c922ec-52a3-4b81-b358-62a41b874da6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:36.626377 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c922ec-52a3-4b81-b358-62a41b874da6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:13:36.626679 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c922ec-52a3-4b81-b358-62a41b874da6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:36.628249 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c922ec-52a3-4b81-b358-62a41b874da6 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:36.640105 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1353/2710] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:13:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:13:36.644315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5a3bd3-fc3c-4b42-bb06-77e1e3aa450c None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:36.653663 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5a3bd3-fc3c-4b42-bb06-77e1e3aa450c None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:36.658444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24a8317-34e7-4ec7-a48f-8d7c52075654 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:36.659028 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24a8317-34e7-4ec7-a48f-8d7c52075654 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:36.659305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24a8317-34e7-4ec7-a48f-8d7c52075654 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:36.659598 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24a8317-34e7-4ec7-a48f-8d7c52075654 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:36.660230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24a8317-34e7-4ec7-a48f-8d7c52075654 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:36.662161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24a8317-34e7-4ec7-a48f-8d7c52075654 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:36.723671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1358/2711] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:13:36 2022] POST /identity/v3/projects => generated 375 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:13:36.727833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c922ec-52a3-4b81-b358-62a41b874da6 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:36.738604 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c922ec-52a3-4b81-b358-62a41b874da6 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:36.796453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d8b532-a037-4289-ac20-be6618ac4847 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:36.797053 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d8b532-a037-4289-ac20-be6618ac4847 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:36.797358 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d8b532-a037-4289-ac20-be6618ac4847 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:36.797628 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d8b532-a037-4289-ac20-be6618ac4847 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:36.798260 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d8b532-a037-4289-ac20-be6618ac4847 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:36.800130 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d8b532-a037-4289-ac20-be6618ac4847 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:36.889123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1354/2712] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:13:36 2022] POST /identity/v3/users => generated 288 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:36.893249 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24a8317-34e7-4ec7-a48f-8d7c52075654 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:36.903416 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24a8317-34e7-4ec7-a48f-8d7c52075654 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:36.908454 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c28e0d-a1c1-4231-adb8-9ec8312280f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:36.909039 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c28e0d-a1c1-4231-adb8-9ec8312280f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:36.909350 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c28e0d-a1c1-4231-adb8-9ec8312280f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:36.909625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c28e0d-a1c1-4231-adb8-9ec8312280f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:36.910375 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c28e0d-a1c1-4231-adb8-9ec8312280f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:36.911962 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c28e0d-a1c1-4231-adb8-9ec8312280f5 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:36.922261 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1359/2713] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:13:36.926268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d8b532-a037-4289-ac20-be6618ac4847 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:36.936881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d8b532-a037-4289-ac20-be6618ac4847 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:36.941565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbae84a-68cc-42b4-aa8a-caf9d8fbf388 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:36.942161 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbae84a-68cc-42b4-aa8a-caf9d8fbf388 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:36.942444 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbae84a-68cc-42b4-aa8a-caf9d8fbf388 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:36.942717 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbae84a-68cc-42b4-aa8a-caf9d8fbf388 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:36.943364 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bbae84a-68cc-42b4-aa8a-caf9d8fbf388 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:36.944953 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bbae84a-68cc-42b4-aa8a-caf9d8fbf388 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:36.956174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1355/2714] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:36.960477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c28e0d-a1c1-4231-adb8-9ec8312280f5 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:36.970513 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c28e0d-a1c1-4231-adb8-9ec8312280f5 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:36.975116 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2897436d-d2c9-4ecc-ba66-170b39dd2617 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:36.975750 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2897436d-d2c9-4ecc-ba66-170b39dd2617 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:36.976028 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2897436d-d2c9-4ecc-ba66-170b39dd2617 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:36.976302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2897436d-d2c9-4ecc-ba66-170b39dd2617 admin admin] PATH_INFO: `/v3/projects/fb2a8002d6bd4e7f9d2484cfd221dae2/users/b1aba812626e4a54aa4c823abe8018fb/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:36.990575 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2897436d-d2c9-4ecc-ba66-170b39dd2617 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb2a8002d6bd4e7f9d2484cfd221dae2, user_id=b1aba812626e4a54aa4c823abe8018fb, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:36.993523 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2897436d-d2c9-4ecc-ba66-170b39dd2617 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:37.105951 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1360/2715] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:13:36 2022] PUT /identity/v3/projects/fb2a8002d6bd4e7f9d2484cfd221dae2/users/b1aba812626e4a54aa4c823abe8018fb/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:37.122690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70bb8e3-b373-4efb-bcac-2593793dbe86 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:37.122981 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70bb8e3-b373-4efb-bcac-2593793dbe86 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:37.123288 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70bb8e3-b373-4efb-bcac-2593793dbe86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:37.151409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-c70bb8e3-b373-4efb-bcac-2593793dbe86 None None] MFA Rules not processed for user `b1aba812626e4a54aa4c823abe8018fb`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:37.152659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-c70bb8e3-b373-4efb-bcac-2593793dbe86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:37.211800 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:37.217045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1356/2716] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:37 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:13:37.227640 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2897436d-d2c9-4ecc-ba66-170b39dd2617 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:37.229360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-2897436d-d2c9-4ecc-ba66-170b39dd2617 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:37.253833 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:37.308296 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2897436d-d2c9-4ecc-ba66-170b39dd2617 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:37.312072 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9712b3bf-488c-445c-9193-556f6424da3a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:37.314093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9712b3bf-488c-445c-9193-556f6424da3a service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:37.314372 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9712b3bf-488c-445c-9193-556f6424da3a service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:37.314644 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9712b3bf-488c-445c-9193-556f6424da3a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:37.329909 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9712b3bf-488c-445c-9193-556f6424da3a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:37.331383 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9712b3bf-488c-445c-9193-556f6424da3a service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:37.341802 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1361/2717] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:37 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:13:47.264334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:47.351040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=26 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:50.792488 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70bb8e3-b373-4efb-bcac-2593793dbe86 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:50.794062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-c70bb8e3-b373-4efb-bcac-2593793dbe86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:50.851457 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70bb8e3-b373-4efb-bcac-2593793dbe86 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:50.855080 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e825f7a-4541-4be5-9902-5094d1c6a285 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:50.855688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e825f7a-4541-4be5-9902-5094d1c6a285 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:50.855966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e825f7a-4541-4be5-9902-5094d1c6a285 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:50.856384 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e825f7a-4541-4be5-9902-5094d1c6a285 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:50.858173 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8e825f7a-4541-4be5-9902-5094d1c6a285 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:50.916552 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e825f7a-4541-4be5-9902-5094d1c6a285 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:50.918027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e825f7a-4541-4be5-9902-5094d1c6a285 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:50.927794 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1357/2718] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:50 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:13:51.883471 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9712b3bf-488c-445c-9193-556f6424da3a service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:51.885043 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9712b3bf-488c-445c-9193-556f6424da3a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:52.007357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9712b3bf-488c-445c-9193-556f6424da3a service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:52.013268 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fe6bf1-51e3-489b-a777-1fecd4fa8deb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:52.013810 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fe6bf1-51e3-489b-a777-1fecd4fa8deb service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:52.014091 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fe6bf1-51e3-489b-a777-1fecd4fa8deb service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:52.014443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fe6bf1-51e3-489b-a777-1fecd4fa8deb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:52.026593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e5fe6bf1-51e3-489b-a777-1fecd4fa8deb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:52.048113 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fe6bf1-51e3-489b-a777-1fecd4fa8deb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:52.051255 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fe6bf1-51e3-489b-a777-1fecd4fa8deb service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:52.063586 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1362/2719] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:13:53.968299 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e825f7a-4541-4be5-9902-5094d1c6a285 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:54.030596 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e825f7a-4541-4be5-9902-5094d1c6a285 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:54.035175 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e117e8a-cacd-4cf3-82d1-94b0f4f9c4e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:54.035806 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e117e8a-cacd-4cf3-82d1-94b0f4f9c4e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:54.036089 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e117e8a-cacd-4cf3-82d1-94b0f4f9c4e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:54.036365 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e117e8a-cacd-4cf3-82d1-94b0f4f9c4e8 admin admin] PATH_INFO: `/v3/users/b1aba812626e4a54aa4c823abe8018fb` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:54.037544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e117e8a-cacd-4cf3-82d1-94b0f4f9c4e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b1aba812626e4a54aa4c823abe8018fb)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:54.039505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e117e8a-cacd-4cf3-82d1-94b0f4f9c4e8 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:54.192950 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5e117e8a-cacd-4cf3-82d1-94b0f4f9c4e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b1aba812626e4a54aa4c823abe8018fb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:54.193629 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5e117e8a-cacd-4cf3-82d1-94b0f4f9c4e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b1aba812626e4a54aa4c823abe8018fb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:54.248700 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5e117e8a-cacd-4cf3-82d1-94b0f4f9c4e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b1aba812626e4a54aa4c823abe8018fb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:54.251476 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:54.263442 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5e117e8a-cacd-4cf3-82d1-94b0f4f9c4e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b1aba812626e4a54aa4c823abe8018fb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:54.264037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5e117e8a-cacd-4cf3-82d1-94b0f4f9c4e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b1aba812626e4a54aa4c823abe8018fb'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:54.277111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1358/2720] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:13:53 2022] DELETE /identity/v3/users/b1aba812626e4a54aa4c823abe8018fb => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:54.284946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180ac971-a74e-4ef4-b951-b234a742de05 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:54.284946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180ac971-a74e-4ef4-b951-b234a742de05 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:54.284946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180ac971-a74e-4ef4-b951-b234a742de05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:54.303565 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-180ac971-a74e-4ef4-b951-b234a742de05 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:54.305286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-180ac971-a74e-4ef4-b951-b234a742de05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:54.368550 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1363/2721] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:54.379977 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e117e8a-cacd-4cf3-82d1-94b0f4f9c4e8 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:54.381494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-5e117e8a-cacd-4cf3-82d1-94b0f4f9c4e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:54.448524 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e117e8a-cacd-4cf3-82d1-94b0f4f9c4e8 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:54.452867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2975a929-e9e2-4925-812e-53efb2997534 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:54.453509 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2975a929-e9e2-4925-812e-53efb2997534 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:54.453791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2975a929-e9e2-4925-812e-53efb2997534 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:54.454066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2975a929-e9e2-4925-812e-53efb2997534 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:54.464988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2975a929-e9e2-4925-812e-53efb2997534 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:54.467235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2975a929-e9e2-4925-812e-53efb2997534 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:54.477246 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1359/2722] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:13:54.975778 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180ac971-a74e-4ef4-b951-b234a742de05 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:54.977319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-180ac971-a74e-4ef4-b951-b234a742de05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:54.991228 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180ac971-a74e-4ef4-b951-b234a742de05 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:54.996183 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16af13b-4fdf-407e-be42-a1dc38afca9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:55.001435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16af13b-4fdf-407e-be42-a1dc38afca9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:55.001435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16af13b-4fdf-407e-be42-a1dc38afca9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:55.001435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16af13b-4fdf-407e-be42-a1dc38afca9c admin admin] PATH_INFO: `/v3/projects/fb2a8002d6bd4e7f9d2484cfd221dae2` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:55.001435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16af13b-4fdf-407e-be42-a1dc38afca9c admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb2a8002d6bd4e7f9d2484cfd221dae2)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:55.001435 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16af13b-4fdf-407e-be42-a1dc38afca9c admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:55.129320 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e16af13b-4fdf-407e-be42-a1dc38afca9c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb2a8002d6bd4e7f9d2484cfd221dae2', 'request_id': 'req-e16af13b-4fdf-407e-be42-a1dc38afca9c'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:55.133354 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:55.138686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e16af13b-4fdf-407e-be42-a1dc38afca9c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb2a8002d6bd4e7f9d2484cfd221dae2', 'request_id': 'req-e16af13b-4fdf-407e-be42-a1dc38afca9c'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:55.142070 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e16af13b-4fdf-407e-be42-a1dc38afca9c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb2a8002d6bd4e7f9d2484cfd221dae2', 'request_id': 'req-e16af13b-4fdf-407e-be42-a1dc38afca9c'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:55.235726 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e16af13b-4fdf-407e-be42-a1dc38afca9c admin admin] The token cache is being invalidate because project fb2a8002d6bd4e7f9d2484cfd221dae2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:13:55.236178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e16af13b-4fdf-407e-be42-a1dc38afca9c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:55.237195 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e16af13b-4fdf-407e-be42-a1dc38afca9c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:13:55.238676 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1364/2723] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:13:54 2022] DELETE /identity/v3/projects/fb2a8002d6bd4e7f9d2484cfd221dae2 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:55.243753 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a130ee-e2a6-4829-93c2-acadefc1f84d None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:55.244262 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a130ee-e2a6-4829-93c2-acadefc1f84d None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:55.244362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a130ee-e2a6-4829-93c2-acadefc1f84d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:55.264819 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-28a130ee-e2a6-4829-93c2-acadefc1f84d None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:55.266220 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-28a130ee-e2a6-4829-93c2-acadefc1f84d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:55.324224 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1360/2724] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:55.333166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be3b0a1-7efb-4658-8f4f-905b6b3e5893 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:55.333666 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be3b0a1-7efb-4658-8f4f-905b6b3e5893 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:55.333771 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be3b0a1-7efb-4658-8f4f-905b6b3e5893 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:55.353908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-7be3b0a1-7efb-4658-8f4f-905b6b3e5893 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:55.355093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7be3b0a1-7efb-4658-8f4f-905b6b3e5893 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:55.363686 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1365/2725] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:55.367859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a130ee-e2a6-4829-93c2-acadefc1f84d None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:55.377707 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a130ee-e2a6-4829-93c2-acadefc1f84d None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:55.382341 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52821a4d-5a68-44f4-a7e6-cb0aaf331060 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:55.382873 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52821a4d-5a68-44f4-a7e6-cb0aaf331060 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:55.383150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52821a4d-5a68-44f4-a7e6-cb0aaf331060 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:55.383422 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52821a4d-5a68-44f4-a7e6-cb0aaf331060 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:55.384071 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52821a4d-5a68-44f4-a7e6-cb0aaf331060 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:13:55.384375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52821a4d-5a68-44f4-a7e6-cb0aaf331060 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:55.385999 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52821a4d-5a68-44f4-a7e6-cb0aaf331060 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:55.397215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1361/2726] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:13:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:13:55.401241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be3b0a1-7efb-4658-8f4f-905b6b3e5893 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:55.410696 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be3b0a1-7efb-4658-8f4f-905b6b3e5893 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:55.415233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71d736d-daeb-4c72-95e4-8b4b172cd727 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:55.415812 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71d736d-daeb-4c72-95e4-8b4b172cd727 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:55.416085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71d736d-daeb-4c72-95e4-8b4b172cd727 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:55.416352 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71d736d-daeb-4c72-95e4-8b4b172cd727 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:55.416980 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71d736d-daeb-4c72-95e4-8b4b172cd727 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:55.418931 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71d736d-daeb-4c72-95e4-8b4b172cd727 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:55.495703 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1366/2727] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:13:55 2022] POST /identity/v3/projects => generated 389 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:13:55.500201 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52821a4d-5a68-44f4-a7e6-cb0aaf331060 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:55.510330 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52821a4d-5a68-44f4-a7e6-cb0aaf331060 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:55.566990 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ccf099-d2a1-4bb4-9cc2-e521f5ac8da1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:55.567617 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ccf099-d2a1-4bb4-9cc2-e521f5ac8da1 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:55.567898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ccf099-d2a1-4bb4-9cc2-e521f5ac8da1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:55.568172 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ccf099-d2a1-4bb4-9cc2-e521f5ac8da1 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:55.568803 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ccf099-d2a1-4bb4-9cc2-e521f5ac8da1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:55.570738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ccf099-d2a1-4bb4-9cc2-e521f5ac8da1 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:55.580985 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:13:55.665669 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1362/2728] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:13:55 2022] POST /identity/v3/users => generated 295 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:13:55.669960 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71d736d-daeb-4c72-95e4-8b4b172cd727 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:55.680814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71d736d-daeb-4c72-95e4-8b4b172cd727 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:55.686060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9345747-d9b4-4523-9f87-a34d2501e0a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:55.686683 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9345747-d9b4-4523-9f87-a34d2501e0a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:55.686963 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9345747-d9b4-4523-9f87-a34d2501e0a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:55.687280 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9345747-d9b4-4523-9f87-a34d2501e0a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:55.687920 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9345747-d9b4-4523-9f87-a34d2501e0a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:55.689540 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9345747-d9b4-4523-9f87-a34d2501e0a9 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:55.699444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1367/2729] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:55 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:55.703941 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ccf099-d2a1-4bb4-9cc2-e521f5ac8da1 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:55.713792 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ccf099-d2a1-4bb4-9cc2-e521f5ac8da1 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:55.718520 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07c825b-c5c1-4a33-b791-b640d458f987 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:55.719105 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07c825b-c5c1-4a33-b791-b640d458f987 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:55.719379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07c825b-c5c1-4a33-b791-b640d458f987 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:55.719646 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07c825b-c5c1-4a33-b791-b640d458f987 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:55.720308 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07c825b-c5c1-4a33-b791-b640d458f987 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:55.721969 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07c825b-c5c1-4a33-b791-b640d458f987 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:55.732978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1363/2730] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:13:55 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:13:55.737166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9345747-d9b4-4523-9f87-a34d2501e0a9 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:55.747347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9345747-d9b4-4523-9f87-a34d2501e0a9 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:55.752627 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c00b4f-ee56-4677-b224-c9c49201536b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:55.753292 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c00b4f-ee56-4677-b224-c9c49201536b admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:55.753570 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c00b4f-ee56-4677-b224-c9c49201536b admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:55.753856 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c00b4f-ee56-4677-b224-c9c49201536b admin admin] PATH_INFO: `/v3/projects/45844a6d610a4549ab7b4a200a111a9f/users/a8da12374eb8481598219b9f818b8901/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:55.768375 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c00b4f-ee56-4677-b224-c9c49201536b admin admin] RBAC: Authorizing `identity:create_grant(project_id=45844a6d610a4549ab7b4a200a111a9f, user_id=a8da12374eb8481598219b9f818b8901, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:55.771040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c00b4f-ee56-4677-b224-c9c49201536b admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:55.860576 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1368/2731] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:13:55 2022] PUT /identity/v3/projects/45844a6d610a4549ab7b4a200a111a9f/users/a8da12374eb8481598219b9f818b8901/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:13:55.925790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537f7d09-b906-48eb-8a1e-bd1b6a2391c2 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:55.926013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537f7d09-b906-48eb-8a1e-bd1b6a2391c2 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:55.926256 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537f7d09-b906-48eb-8a1e-bd1b6a2391c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:55.954704 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-537f7d09-b906-48eb-8a1e-bd1b6a2391c2 None None] MFA Rules not processed for user `a8da12374eb8481598219b9f818b8901`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:13:55.955918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-537f7d09-b906-48eb-8a1e-bd1b6a2391c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:56.022785 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1364/2732] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:13:55 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:13:56.033802 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c00b4f-ee56-4677-b224-c9c49201536b admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:13:56.035621 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-17c00b4f-ee56-4677-b224-c9c49201536b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:13:56.095994 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c00b4f-ee56-4677-b224-c9c49201536b admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:13:56.099657 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49724723-76d3-4310-81b9-5bc1b3885b0f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:13:56.100263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49724723-76d3-4310-81b9-5bc1b3885b0f service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:13:56.100542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49724723-76d3-4310-81b9-5bc1b3885b0f service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:13:56.100814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49724723-76d3-4310-81b9-5bc1b3885b0f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:13:56.109105 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49724723-76d3-4310-81b9-5bc1b3885b0f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:13:56.110544 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49724723-76d3-4310-81b9-5bc1b3885b0f service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:13:56.119453 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1369/2733] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:13:56 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:06.019807 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:INSERT=1,keystone:SELECT=39 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:06.115230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:DELETE=2,keystone:SELECT=58,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:06.385946 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537f7d09-b906-48eb-8a1e-bd1b6a2391c2 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:06.387550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-537f7d09-b906-48eb-8a1e-bd1b6a2391c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:06.476737 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537f7d09-b906-48eb-8a1e-bd1b6a2391c2 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:06.483587 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3494794-330a-4ecc-accf-7de3034b6d77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:06.484514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3494794-330a-4ecc-accf-7de3034b6d77 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:06.484957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3494794-330a-4ecc-accf-7de3034b6d77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:06.485390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3494794-330a-4ecc-accf-7de3034b6d77 admin admin] PATH_INFO: `/v3/users/4dbfca17abc04d4580b2997896a94776` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:06.486543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3494794-330a-4ecc-accf-7de3034b6d77 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4dbfca17abc04d4580b2997896a94776)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:06.488501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3494794-330a-4ecc-accf-7de3034b6d77 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:06.675146 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a3494794-330a-4ecc-accf-7de3034b6d77 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4dbfca17abc04d4580b2997896a94776'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:06.675848 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a3494794-330a-4ecc-accf-7de3034b6d77 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4dbfca17abc04d4580b2997896a94776'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:06.742112 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a3494794-330a-4ecc-accf-7de3034b6d77 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4dbfca17abc04d4580b2997896a94776'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:06.743611 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49724723-76d3-4310-81b9-5bc1b3885b0f service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:06.745483 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-49724723-76d3-4310-81b9-5bc1b3885b0f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:06.755193 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a3494794-330a-4ecc-accf-7de3034b6d77 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4dbfca17abc04d4580b2997896a94776'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:06.756410 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-a3494794-330a-4ecc-accf-7de3034b6d77 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4dbfca17abc04d4580b2997896a94776'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:06.772229 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1365/2734] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:14:06 2022] DELETE /identity/v3/users/4dbfca17abc04d4580b2997896a94776 => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:06.776371 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b4d24f-9d9f-4837-89cb-ac3c5bc10559 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:06.776661 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b4d24f-9d9f-4837-89cb-ac3c5bc10559 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:06.776938 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b4d24f-9d9f-4837-89cb-ac3c5bc10559 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:06.802343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-20b4d24f-9d9f-4837-89cb-ac3c5bc10559 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:06.804059 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-20b4d24f-9d9f-4837-89cb-ac3c5bc10559 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:06.806702 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49724723-76d3-4310-81b9-5bc1b3885b0f service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:06.866443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdb0046-df96-40a2-9ae8-dd35c2ee507a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:06.867059 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdb0046-df96-40a2-9ae8-dd35c2ee507a service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:06.867339 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdb0046-df96-40a2-9ae8-dd35c2ee507a service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:06.867615 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdb0046-df96-40a2-9ae8-dd35c2ee507a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:06.869690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7fdb0046-df96-40a2-9ae8-dd35c2ee507a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:06.882162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1366/2735] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:14:06.893957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b4d24f-9d9f-4837-89cb-ac3c5bc10559 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:06.896294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-20b4d24f-9d9f-4837-89cb-ac3c5bc10559 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:06.910375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b4d24f-9d9f-4837-89cb-ac3c5bc10559 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:06.914073 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eee1163-810c-4e58-bbe1-3881106551f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:06.914714 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eee1163-810c-4e58-bbe1-3881106551f1 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:06.915010 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eee1163-810c-4e58-bbe1-3881106551f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:06.915294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eee1163-810c-4e58-bbe1-3881106551f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:06.925657 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eee1163-810c-4e58-bbe1-3881106551f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:06.927084 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eee1163-810c-4e58-bbe1-3881106551f1 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:06.937460 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1367/2736] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:14:06.939388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdb0046-df96-40a2-9ae8-dd35c2ee507a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:06.941340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdb0046-df96-40a2-9ae8-dd35c2ee507a service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:06.953485 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1370/2737] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:07.515641 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eee1163-810c-4e58-bbe1-3881106551f1 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:07.515641 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-5eee1163-810c-4e58-bbe1-3881106551f1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:07.539153 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eee1163-810c-4e58-bbe1-3881106551f1 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:07.549665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da8f4c4-6965-4ca3-8e0c-09bdaba7307c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:07.549665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da8f4c4-6965-4ca3-8e0c-09bdaba7307c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:07.549665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da8f4c4-6965-4ca3-8e0c-09bdaba7307c admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:07.549665 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da8f4c4-6965-4ca3-8e0c-09bdaba7307c admin admin] PATH_INFO: `/v3/projects/9316dc6b766147b4ac43791e2714f2e5` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:07.550041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da8f4c4-6965-4ca3-8e0c-09bdaba7307c admin admin] RBAC: Authorizing `identity:delete_project(project_id=9316dc6b766147b4ac43791e2714f2e5)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:07.550386 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da8f4c4-6965-4ca3-8e0c-09bdaba7307c admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:07.551882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:07.669104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8da8f4c4-6965-4ca3-8e0c-09bdaba7307c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9316dc6b766147b4ac43791e2714f2e5', 'request_id': 'req-8da8f4c4-6965-4ca3-8e0c-09bdaba7307c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:07.675390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8da8f4c4-6965-4ca3-8e0c-09bdaba7307c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9316dc6b766147b4ac43791e2714f2e5', 'request_id': 'req-8da8f4c4-6965-4ca3-8e0c-09bdaba7307c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:07.726832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8da8f4c4-6965-4ca3-8e0c-09bdaba7307c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9316dc6b766147b4ac43791e2714f2e5', 'request_id': 'req-8da8f4c4-6965-4ca3-8e0c-09bdaba7307c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:07.742600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8da8f4c4-6965-4ca3-8e0c-09bdaba7307c admin admin] The token cache is being invalidate because project 9316dc6b766147b4ac43791e2714f2e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:14:07.745512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8da8f4c4-6965-4ca3-8e0c-09bdaba7307c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:07.745512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8da8f4c4-6965-4ca3-8e0c-09bdaba7307c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:07.745689 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1368/2738] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:14:07 2022] DELETE /identity/v3/projects/9316dc6b766147b4ac43791e2714f2e5 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:07.750933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042c6101-dfaa-47ea-b4dc-477ce7fb9aff None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:07.751499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042c6101-dfaa-47ea-b4dc-477ce7fb9aff None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:07.751499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042c6101-dfaa-47ea-b4dc-477ce7fb9aff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:07.781513 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-042c6101-dfaa-47ea-b4dc-477ce7fb9aff None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:07.782924 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-042c6101-dfaa-47ea-b4dc-477ce7fb9aff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:07.851683 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1371/2739] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:14:07.860644 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd999a1-d65e-4167-807f-161caa42e555 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:07.861087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd999a1-d65e-4167-807f-161caa42e555 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:07.861396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd999a1-d65e-4167-807f-161caa42e555 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:07.881166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042c6101-dfaa-47ea-b4dc-477ce7fb9aff None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:07.882656 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-042c6101-dfaa-47ea-b4dc-477ce7fb9aff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:07.887634 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-efd999a1-d65e-4167-807f-161caa42e555 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:07.890062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-efd999a1-d65e-4167-807f-161caa42e555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:07.897259 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042c6101-dfaa-47ea-b4dc-477ce7fb9aff None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:07.900580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1369/2740] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:07.904700 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee60175-313e-4ade-8eb8-a36ab97e9312 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:07.905364 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee60175-313e-4ade-8eb8-a36ab97e9312 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:07.905607 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd999a1-d65e-4167-807f-161caa42e555 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:07.905717 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee60175-313e-4ade-8eb8-a36ab97e9312 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:07.905893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee60175-313e-4ade-8eb8-a36ab97e9312 admin admin] PATH_INFO: `/v3/users/2c1fe639fb5847fe9bbe1303b8688215` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:07.907185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee60175-313e-4ade-8eb8-a36ab97e9312 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c1fe639fb5847fe9bbe1303b8688215)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:07.908996 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee60175-313e-4ade-8eb8-a36ab97e9312 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:07.910719 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:07.916839 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd999a1-d65e-4167-807f-161caa42e555 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:07.921833 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26e05cc-b666-4fc1-954a-552fffa27fc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:07.922440 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26e05cc-b666-4fc1-954a-552fffa27fc2 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:07.922720 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26e05cc-b666-4fc1-954a-552fffa27fc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:07.925350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26e05cc-b666-4fc1-954a-552fffa27fc2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:07.925350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26e05cc-b666-4fc1-954a-552fffa27fc2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:14:07.925350 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26e05cc-b666-4fc1-954a-552fffa27fc2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:07.925707 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26e05cc-b666-4fc1-954a-552fffa27fc2 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:07.938291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1370/2741] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:14:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:14:07.942649 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26e05cc-b666-4fc1-954a-552fffa27fc2 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:07.953352 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26e05cc-b666-4fc1-954a-552fffa27fc2 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:07.959400 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65da07bf-a3b6-4c95-ae2c-b36f90c1caf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:07.960005 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65da07bf-a3b6-4c95-ae2c-b36f90c1caf4 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:07.960284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65da07bf-a3b6-4c95-ae2c-b36f90c1caf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:07.961011 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65da07bf-a3b6-4c95-ae2c-b36f90c1caf4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:07.961238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65da07bf-a3b6-4c95-ae2c-b36f90c1caf4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:07.963122 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65da07bf-a3b6-4c95-ae2c-b36f90c1caf4 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:08.061968 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1371/2742] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:14:07 2022] POST /identity/v3/projects => generated 383 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:14:08.066015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65da07bf-a3b6-4c95-ae2c-b36f90c1caf4 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:08.103823 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-8ee60175-313e-4ade-8eb8-a36ab97e9312 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c1fe639fb5847fe9bbe1303b8688215'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:08.139960 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65da07bf-a3b6-4c95-ae2c-b36f90c1caf4 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:08.144775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef91791-9dc7-491b-8fb3-8b1e498c3642 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:08.145427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef91791-9dc7-491b-8fb3-8b1e498c3642 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:08.145710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef91791-9dc7-491b-8fb3-8b1e498c3642 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:08.145985 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef91791-9dc7-491b-8fb3-8b1e498c3642 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:08.146622 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef91791-9dc7-491b-8fb3-8b1e498c3642 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:08.148588 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef91791-9dc7-491b-8fb3-8b1e498c3642 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:08.174441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-8ee60175-313e-4ade-8eb8-a36ab97e9312 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c1fe639fb5847fe9bbe1303b8688215'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:08.187910 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-8ee60175-313e-4ade-8eb8-a36ab97e9312 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c1fe639fb5847fe9bbe1303b8688215'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:08.188725 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-8ee60175-313e-4ade-8eb8-a36ab97e9312 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c1fe639fb5847fe9bbe1303b8688215'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:08.196305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-8ee60175-313e-4ade-8eb8-a36ab97e9312 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c1fe639fb5847fe9bbe1303b8688215'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:08.202475 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1372/2743] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:14:07 2022] DELETE /identity/v3/users/2c1fe639fb5847fe9bbe1303b8688215 => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:14:08.206897 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4e80b3-262b-44b3-adda-3ace64a5c466 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:08.207225 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4e80b3-262b-44b3-adda-3ace64a5c466 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:08.207534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4e80b3-262b-44b3-adda-3ace64a5c466 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:08.222623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1372/2744] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:14:08 2022] POST /identity/v3/users => generated 292 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:14:08.227500 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef91791-9dc7-491b-8fb3-8b1e498c3642 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:08.227963 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-ff4e80b3-262b-44b3-adda-3ace64a5c466 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:08.229240 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ff4e80b3-262b-44b3-adda-3ace64a5c466 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:08.229565 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-0ef91791-9dc7-491b-8fb3-8b1e498c3642 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:08.245399 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef91791-9dc7-491b-8fb3-8b1e498c3642 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:08.297383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:08.318312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1373/2745] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:14:08.331064 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4e80b3-262b-44b3-adda-3ace64a5c466 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:08.332627 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ff4e80b3-262b-44b3-adda-3ace64a5c466 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:08.341580 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992424d2-b8db-4d91-829a-4fb0038d4f69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:08.342447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992424d2-b8db-4d91-829a-4fb0038d4f69 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:08.342748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992424d2-b8db-4d91-829a-4fb0038d4f69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:08.343187 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992424d2-b8db-4d91-829a-4fb0038d4f69 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:08.344008 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992424d2-b8db-4d91-829a-4fb0038d4f69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:08.345909 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992424d2-b8db-4d91-829a-4fb0038d4f69 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:08.364123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1373/2746] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:08 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:14:08.366348 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992424d2-b8db-4d91-829a-4fb0038d4f69 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:08.376907 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992424d2-b8db-4d91-829a-4fb0038d4f69 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:08.382662 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52608716-d137-4b6a-8b65-637815fac29b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:08.383291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52608716-d137-4b6a-8b65-637815fac29b admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:08.383563 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52608716-d137-4b6a-8b65-637815fac29b admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:08.383840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52608716-d137-4b6a-8b65-637815fac29b admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:08.384941 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52608716-d137-4b6a-8b65-637815fac29b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:08.388848 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52608716-d137-4b6a-8b65-637815fac29b admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:08.403070 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1374/2747] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:08 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:08.407980 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52608716-d137-4b6a-8b65-637815fac29b admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:08.410744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4e80b3-262b-44b3-adda-3ace64a5c466 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:08.415819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0813616-df80-40b6-ab47-c1e2192c8f88 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:08.416866 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0813616-df80-40b6-ab47-c1e2192c8f88 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:08.417464 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0813616-df80-40b6-ab47-c1e2192c8f88 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:08.418000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0813616-df80-40b6-ab47-c1e2192c8f88 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:08.419484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52608716-d137-4b6a-8b65-637815fac29b admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:08.431458 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abb39ea-1150-4521-b1ac-88e92014ee44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:08.432185 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abb39ea-1150-4521-b1ac-88e92014ee44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:08.432474 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abb39ea-1150-4521-b1ac-88e92014ee44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:08.432746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abb39ea-1150-4521-b1ac-88e92014ee44 admin admin] PATH_INFO: `/v3/projects/027792a6118140099aaf522ca15bb07e/users/9b5efa37b94b47949344d00ff88be4fb/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:08.437960 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0813616-df80-40b6-ab47-c1e2192c8f88 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:08.439889 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0813616-df80-40b6-ab47-c1e2192c8f88 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:08.451592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1374/2748] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:14:08.456379 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abb39ea-1150-4521-b1ac-88e92014ee44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=027792a6118140099aaf522ca15bb07e, user_id=9b5efa37b94b47949344d00ff88be4fb, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:08.462940 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abb39ea-1150-4521-b1ac-88e92014ee44 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:08.575715 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1375/2749] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:14:08 2022] PUT /identity/v3/projects/027792a6118140099aaf522ca15bb07e/users/9b5efa37b94b47949344d00ff88be4fb/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:08.582122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b08a881-8135-4a3a-8e0c-c47f8a0671a4 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:08.582441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b08a881-8135-4a3a-8e0c-c47f8a0671a4 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:08.582706 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b08a881-8135-4a3a-8e0c-c47f8a0671a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:08.604620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-4b08a881-8135-4a3a-8e0c-c47f8a0671a4 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:08.605893 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4b08a881-8135-4a3a-8e0c-c47f8a0671a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:08.626240 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:08.669006 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1375/2750] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:14:08.679137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abb39ea-1150-4521-b1ac-88e92014ee44 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:08.680865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-6abb39ea-1150-4521-b1ac-88e92014ee44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:08.740119 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abb39ea-1150-4521-b1ac-88e92014ee44 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:08.744048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ede71cd-c45f-4534-a53f-60c028e7a9c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:08.744658 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ede71cd-c45f-4534-a53f-60c028e7a9c4 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:08.744942 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ede71cd-c45f-4534-a53f-60c028e7a9c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:08.745241 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ede71cd-c45f-4534-a53f-60c028e7a9c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:08.755101 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ede71cd-c45f-4534-a53f-60c028e7a9c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:08.759534 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ede71cd-c45f-4534-a53f-60c028e7a9c4 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:08.766584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1376/2751] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:09.028424 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b08a881-8135-4a3a-8e0c-c47f8a0671a4 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:09.032669 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4b08a881-8135-4a3a-8e0c-c47f8a0671a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:09.045436 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b08a881-8135-4a3a-8e0c-c47f8a0671a4 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:09.066868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54891c6-861b-4c10-9212-d21f8fa59cc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:09.067934 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54891c6-861b-4c10-9212-d21f8fa59cc7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:09.068243 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54891c6-861b-4c10-9212-d21f8fa59cc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:09.068520 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54891c6-861b-4c10-9212-d21f8fa59cc7 admin admin] PATH_INFO: `/v3/projects/66afded63f464752aa18426be3a437ac` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:09.069705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54891c6-861b-4c10-9212-d21f8fa59cc7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=66afded63f464752aa18426be3a437ac)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:09.071620 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54891c6-861b-4c10-9212-d21f8fa59cc7 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:09.202680 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e54891c6-861b-4c10-9212-d21f8fa59cc7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '66afded63f464752aa18426be3a437ac', 'request_id': 'req-e54891c6-861b-4c10-9212-d21f8fa59cc7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:09.208765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e54891c6-861b-4c10-9212-d21f8fa59cc7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '66afded63f464752aa18426be3a437ac', 'request_id': 'req-e54891c6-861b-4c10-9212-d21f8fa59cc7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:09.215179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e54891c6-861b-4c10-9212-d21f8fa59cc7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '66afded63f464752aa18426be3a437ac', 'request_id': 'req-e54891c6-861b-4c10-9212-d21f8fa59cc7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:09.365014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e54891c6-861b-4c10-9212-d21f8fa59cc7 admin admin] The token cache is being invalidate because project 66afded63f464752aa18426be3a437ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:14:09.365353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e54891c6-861b-4c10-9212-d21f8fa59cc7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:09.366098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-e54891c6-861b-4c10-9212-d21f8fa59cc7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:09.371933 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1376/2752] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:14:09 2022] DELETE /identity/v3/projects/66afded63f464752aa18426be3a437ac => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:09.373049 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ede71cd-c45f-4534-a53f-60c028e7a9c4 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:09.386181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ede71cd-c45f-4534-a53f-60c028e7a9c4 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:09.455900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c53eae-3ce3-4b12-b3de-6b40a04c4951 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:09.456552 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c53eae-3ce3-4b12-b3de-6b40a04c4951 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:09.456834 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c53eae-3ce3-4b12-b3de-6b40a04c4951 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:09.457290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c53eae-3ce3-4b12-b3de-6b40a04c4951 admin admin] PATH_INFO: `/v3/users/cddb4a0de03e4d369cbd1428de5c14f8` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:09.466062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c53eae-3ce3-4b12-b3de-6b40a04c4951 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cddb4a0de03e4d369cbd1428de5c14f8)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:09.468512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c53eae-3ce3-4b12-b3de-6b40a04c4951 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:09.689531 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:09.692394 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-26c53eae-3ce3-4b12-b3de-6b40a04c4951 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cddb4a0de03e4d369cbd1428de5c14f8'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:09.693064 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-26c53eae-3ce3-4b12-b3de-6b40a04c4951 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cddb4a0de03e4d369cbd1428de5c14f8'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:09.763876 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-26c53eae-3ce3-4b12-b3de-6b40a04c4951 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cddb4a0de03e4d369cbd1428de5c14f8'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:09.778263 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-26c53eae-3ce3-4b12-b3de-6b40a04c4951 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cddb4a0de03e4d369cbd1428de5c14f8'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:09.779026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-26c53eae-3ce3-4b12-b3de-6b40a04c4951 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cddb4a0de03e4d369cbd1428de5c14f8'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:09.792652 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1377/2753] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:14:09 2022] DELETE /identity/v3/users/cddb4a0de03e4d369cbd1428de5c14f8 => generated 0 bytes in 420 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:14:10.445154 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54891c6-861b-4c10-9212-d21f8fa59cc7 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:10.447310 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-e54891c6-861b-4c10-9212-d21f8fa59cc7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:10.466844 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54891c6-861b-4c10-9212-d21f8fa59cc7 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:10.558589 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090f835f-0286-497e-8d36-468278a05c86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:10.559106 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090f835f-0286-497e-8d36-468278a05c86 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:10.559385 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090f835f-0286-497e-8d36-468278a05c86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:10.559656 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090f835f-0286-497e-8d36-468278a05c86 admin admin] PATH_INFO: `/v3/projects/da3cd3ece2bf4885892aa6b2afdbd787` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:10.567298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090f835f-0286-497e-8d36-468278a05c86 admin admin] RBAC: Authorizing `identity:delete_project(project_id=da3cd3ece2bf4885892aa6b2afdbd787)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:10.582042 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090f835f-0286-497e-8d36-468278a05c86 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:10.775221 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:10.777245 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-090f835f-0286-497e-8d36-468278a05c86 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'da3cd3ece2bf4885892aa6b2afdbd787', 'request_id': 'req-090f835f-0286-497e-8d36-468278a05c86'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:10.786684 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-090f835f-0286-497e-8d36-468278a05c86 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'da3cd3ece2bf4885892aa6b2afdbd787', 'request_id': 'req-090f835f-0286-497e-8d36-468278a05c86'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:10.813159 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-090f835f-0286-497e-8d36-468278a05c86 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'da3cd3ece2bf4885892aa6b2afdbd787', 'request_id': 'req-090f835f-0286-497e-8d36-468278a05c86'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:10.899409 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-090f835f-0286-497e-8d36-468278a05c86 admin admin] The token cache is being invalidate because project da3cd3ece2bf4885892aa6b2afdbd787 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:14:10.899815 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-090f835f-0286-497e-8d36-468278a05c86 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:10.900612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-090f835f-0286-497e-8d36-468278a05c86 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:10.909291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854c760-fb64-43bc-abfd-1e75327690b4 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:10.910945 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854c760-fb64-43bc-abfd-1e75327690b4 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:10.911456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f854c760-fb64-43bc-abfd-1e75327690b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:10.915254 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1377/2754] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:14:10 2022] DELETE /identity/v3/projects/da3cd3ece2bf4885892aa6b2afdbd787 => generated 0 bytes in 459 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:10.942586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-f854c760-fb64-43bc-abfd-1e75327690b4 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:10.944186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f854c760-fb64-43bc-abfd-1e75327690b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:11.027207 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07790421-c6c0-4c60-a813-1f865c7317c8 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:11.027505 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07790421-c6c0-4c60-a813-1f865c7317c8 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:11.027855 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07790421-c6c0-4c60-a813-1f865c7317c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:11.034309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1378/2755] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:14:11.056312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-07790421-c6c0-4c60-a813-1f865c7317c8 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:11.058067 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-07790421-c6c0-4c60-a813-1f865c7317c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:11.074377 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1378/2756] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:11.076609 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f854c760-fb64-43bc-abfd-1e75327690b4 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:11.093044 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f854c760-fb64-43bc-abfd-1e75327690b4 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:11.100100 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7fba66-5098-45d8-a9c7-162a67d7a860 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:11.100724 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7fba66-5098-45d8-a9c7-162a67d7a860 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:11.101007 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7fba66-5098-45d8-a9c7-162a67d7a860 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:11.101319 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7fba66-5098-45d8-a9c7-162a67d7a860 admin admin] PATH_INFO: `/v3/domains` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:11.101999 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7fba66-5098-45d8-a9c7-162a67d7a860 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81559) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:14:11.102606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7fba66-5098-45d8-a9c7-162a67d7a860 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:11.105583 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7fba66-5098-45d8-a9c7-162a67d7a860 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:11.125835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1379/2757] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:14:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 01:14:11.129644 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07790421-c6c0-4c60-a813-1f865c7317c8 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:11.140049 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07790421-c6c0-4c60-a813-1f865c7317c8 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:11.148593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfaa1955-daec-4c94-ad0a-623d37dcccaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:11.149245 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaa1955-daec-4c94-ad0a-623d37dcccaf admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:11.149527 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaa1955-daec-4c94-ad0a-623d37dcccaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:11.149800 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaa1955-daec-4c94-ad0a-623d37dcccaf admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:11.150457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfaa1955-daec-4c94-ad0a-623d37dcccaf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:11.152363 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfaa1955-daec-4c94-ad0a-623d37dcccaf admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:11.344333 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1379/2758] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:14:11 2022] POST /identity/v3/projects => generated 357 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:14:11.348791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7fba66-5098-45d8-a9c7-162a67d7a860 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:11.367217 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7fba66-5098-45d8-a9c7-162a67d7a860 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:11.437294 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9726b87f-59aa-45d0-8051-f86060b79935 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:11.437467 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9726b87f-59aa-45d0-8051-f86060b79935 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:11.440307 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9726b87f-59aa-45d0-8051-f86060b79935 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:11.440307 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9726b87f-59aa-45d0-8051-f86060b79935 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:11.440307 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9726b87f-59aa-45d0-8051-f86060b79935 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:11.442148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9726b87f-59aa-45d0-8051-f86060b79935 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:11.587199 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1380/2759] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:14:11 2022] POST /identity/v3/users => generated 279 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:14:11.588755 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfaa1955-daec-4c94-ad0a-623d37dcccaf admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:11.602393 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfaa1955-daec-4c94-ad0a-623d37dcccaf admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:11.617740 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18388b13-402a-4e4b-b626-1a98d0eca78b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:11.620266 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18388b13-402a-4e4b-b626-1a98d0eca78b admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:11.620788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18388b13-402a-4e4b-b626-1a98d0eca78b admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:11.621306 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18388b13-402a-4e4b-b626-1a98d0eca78b admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:11.622163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18388b13-402a-4e4b-b626-1a98d0eca78b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:11.625684 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18388b13-402a-4e4b-b626-1a98d0eca78b admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:11.642164 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1380/2760] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:11 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:11.646225 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9726b87f-59aa-45d0-8051-f86060b79935 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:11.657975 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9726b87f-59aa-45d0-8051-f86060b79935 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:11.667871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b872fc8-5d66-48c9-8874-8673017cbf46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:11.668616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b872fc8-5d66-48c9-8874-8673017cbf46 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:11.668994 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b872fc8-5d66-48c9-8874-8673017cbf46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:11.670202 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b872fc8-5d66-48c9-8874-8673017cbf46 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:11.670862 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b872fc8-5d66-48c9-8874-8673017cbf46 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:11.672600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b872fc8-5d66-48c9-8874-8673017cbf46 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:11.684449 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1381/2761] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:11 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:11.689098 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18388b13-402a-4e4b-b626-1a98d0eca78b admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:11.704096 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18388b13-402a-4e4b-b626-1a98d0eca78b admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:11.709312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112cf28b-ca06-42ba-9140-ff98948cd892 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:11.710038 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112cf28b-ca06-42ba-9140-ff98948cd892 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:11.710322 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112cf28b-ca06-42ba-9140-ff98948cd892 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:11.710599 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112cf28b-ca06-42ba-9140-ff98948cd892 admin admin] PATH_INFO: `/v3/projects/b5bfefa807364899ae84d6ae0acf6c89/users/1bfe064e18d542d1a8b587c9ba76e83c/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:11.727103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112cf28b-ca06-42ba-9140-ff98948cd892 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5bfefa807364899ae84d6ae0acf6c89, user_id=1bfe064e18d542d1a8b587c9ba76e83c, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:11.730140 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112cf28b-ca06-42ba-9140-ff98948cd892 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:11.845830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1381/2762] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:14:11 2022] PUT /identity/v3/projects/b5bfefa807364899ae84d6ae0acf6c89/users/1bfe064e18d542d1a8b587c9ba76e83c/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:11.847792 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353ff4f3-fba2-4e17-953b-cd07c2d5c977 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:11.848136 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353ff4f3-fba2-4e17-953b-cd07c2d5c977 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:11.848415 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353ff4f3-fba2-4e17-953b-cd07c2d5c977 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:11.880135 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-353ff4f3-fba2-4e17-953b-cd07c2d5c977 None None] MFA Rules not processed for user `1bfe064e18d542d1a8b587c9ba76e83c`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:11.881436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-353ff4f3-fba2-4e17-953b-cd07c2d5c977 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:11.895923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:11.955496 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1382/2763] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:11 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:11.960180 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112cf28b-ca06-42ba-9140-ff98948cd892 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:12.046906 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112cf28b-ca06-42ba-9140-ff98948cd892 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:12.053331 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fec02d-2e5a-40a3-ae39-6b6d30c5519f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:12.053968 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fec02d-2e5a-40a3-ae39-6b6d30c5519f admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:12.054245 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fec02d-2e5a-40a3-ae39-6b6d30c5519f admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:12.054508 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fec02d-2e5a-40a3-ae39-6b6d30c5519f admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:12.055324 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fec02d-2e5a-40a3-ae39-6b6d30c5519f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:12.057519 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fec02d-2e5a-40a3-ae39-6b6d30c5519f admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:12.194851 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1382/2764] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:14:11 2022] POST /identity/v3/projects => generated 355 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:14:12.199616 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353ff4f3-fba2-4e17-953b-cd07c2d5c977 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:12.220041 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353ff4f3-fba2-4e17-953b-cd07c2d5c977 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:12.299251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee856eb5-e231-4ceb-8b85-79a19caaad6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:12.300119 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee856eb5-e231-4ceb-8b85-79a19caaad6f admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:12.300497 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee856eb5-e231-4ceb-8b85-79a19caaad6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:12.300870 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee856eb5-e231-4ceb-8b85-79a19caaad6f admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:12.301645 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee856eb5-e231-4ceb-8b85-79a19caaad6f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:12.304194 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee856eb5-e231-4ceb-8b85-79a19caaad6f admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:12.400882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1383/2765] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:14:12 2022] POST /identity/v3/users => generated 284 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:14:12.402195 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fec02d-2e5a-40a3-ae39-6b6d30c5519f admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:12.417890 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fec02d-2e5a-40a3-ae39-6b6d30c5519f admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:12.436427 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f42fa9a-7f49-4573-b0e4-affc141a12dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:12.437141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f42fa9a-7f49-4573-b0e4-affc141a12dc admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:12.437430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f42fa9a-7f49-4573-b0e4-affc141a12dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:12.437700 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f42fa9a-7f49-4573-b0e4-affc141a12dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:12.439682 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f42fa9a-7f49-4573-b0e4-affc141a12dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:12.441286 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f42fa9a-7f49-4573-b0e4-affc141a12dc admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:12.455069 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1383/2766] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:12 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:12.459861 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee856eb5-e231-4ceb-8b85-79a19caaad6f admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:12.479178 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee856eb5-e231-4ceb-8b85-79a19caaad6f admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:12.484853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f6b95b-3eee-4184-a7a7-6719bc788ab1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:12.485776 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f6b95b-3eee-4184-a7a7-6719bc788ab1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:12.485900 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f6b95b-3eee-4184-a7a7-6719bc788ab1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:12.486172 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f6b95b-3eee-4184-a7a7-6719bc788ab1 admin admin] PATH_INFO: `/v3/projects/136968e58b1b42289f63983c8d7f4927/users/c50d8f2f03584a7994b238f0dc6d75e8/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:12.504619 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f6b95b-3eee-4184-a7a7-6719bc788ab1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=136968e58b1b42289f63983c8d7f4927, user_id=c50d8f2f03584a7994b238f0dc6d75e8, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:12.509020 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f6b95b-3eee-4184-a7a7-6719bc788ab1 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:12.603157 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1384/2767] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:14:12 2022] PUT /identity/v3/projects/136968e58b1b42289f63983c8d7f4927/users/c50d8f2f03584a7994b238f0dc6d75e8/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:12.605710 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f42fa9a-7f49-4573-b0e4-affc141a12dc admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:12.616875 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f42fa9a-7f49-4573-b0e4-affc141a12dc admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:12.708567 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:12.722781 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23dcda6-74dd-4238-b7be-075ab88a241b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:12.724002 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23dcda6-74dd-4238-b7be-075ab88a241b admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:12.724499 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23dcda6-74dd-4238-b7be-075ab88a241b admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:12.724974 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23dcda6-74dd-4238-b7be-075ab88a241b admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:12.726152 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23dcda6-74dd-4238-b7be-075ab88a241b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:12.728094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23dcda6-74dd-4238-b7be-075ab88a241b admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:12.750957 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1384/2768] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:12 2022] GET /identity/v3/roles => generated 1420 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:12.754938 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f6b95b-3eee-4184-a7a7-6719bc788ab1 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:12.768224 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f6b95b-3eee-4184-a7a7-6719bc788ab1 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:12.773867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92803d1f-5057-4a61-9c84-df3b8df39763 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:12.774792 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92803d1f-5057-4a61-9c84-df3b8df39763 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:12.775035 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92803d1f-5057-4a61-9c84-df3b8df39763 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:12.775309 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92803d1f-5057-4a61-9c84-df3b8df39763 admin admin] PATH_INFO: `/v3/projects/136968e58b1b42289f63983c8d7f4927/users/c50d8f2f03584a7994b238f0dc6d75e8/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:12.777780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92803d1f-5057-4a61-9c84-df3b8df39763 admin admin] RBAC: Authorizing `identity:create_grant(project_id=136968e58b1b42289f63983c8d7f4927, user_id=c50d8f2f03584a7994b238f0dc6d75e8, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:12.783227 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92803d1f-5057-4a61-9c84-df3b8df39763 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:12.814420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1385/2769] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:14:12 2022] PUT /identity/v3/projects/136968e58b1b42289f63983c8d7f4927/users/c50d8f2f03584a7994b238f0dc6d75e8/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:12.826337 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d56723-c0d5-4fcc-9746-4edb2d72de83 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:12.826647 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d56723-c0d5-4fcc-9746-4edb2d72de83 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:12.826923 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d56723-c0d5-4fcc-9746-4edb2d72de83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:12.898727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-53d56723-c0d5-4fcc-9746-4edb2d72de83 None None] MFA Rules not processed for user `c50d8f2f03584a7994b238f0dc6d75e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:12.900103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-53d56723-c0d5-4fcc-9746-4edb2d72de83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:13.037313 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1385/2770] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:12 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:13.049728 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92803d1f-5057-4a61-9c84-df3b8df39763 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:13.052047 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-92803d1f-5057-4a61-9c84-df3b8df39763 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:13.152440 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92803d1f-5057-4a61-9c84-df3b8df39763 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:13.158748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc996f63-ff4f-4794-b605-0338796fa3bd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:13.159698 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc996f63-ff4f-4794-b605-0338796fa3bd service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:13.160301 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc996f63-ff4f-4794-b605-0338796fa3bd service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:13.160728 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc996f63-ff4f-4794-b605-0338796fa3bd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:13.172504 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc996f63-ff4f-4794-b605-0338796fa3bd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:13.174663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc996f63-ff4f-4794-b605-0338796fa3bd service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:13.189624 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1386/2771] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:13 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:13.395014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d56723-c0d5-4fcc-9746-4edb2d72de83 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:13.416958 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d56723-c0d5-4fcc-9746-4edb2d72de83 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:13.427656 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2a2b5c-185f-4667-a037-859c6177e39d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:13.428431 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2a2b5c-185f-4667-a037-859c6177e39d service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:13.429148 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2a2b5c-185f-4667-a037-859c6177e39d service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:13.429148 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2a2b5c-185f-4667-a037-859c6177e39d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:13.522203 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2a2b5c-185f-4667-a037-859c6177e39d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:13.523781 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2a2b5c-185f-4667-a037-859c6177e39d service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:13.536060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1386/2772] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:13 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:15.241678 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc84132-42b2-4bbb-a048-a6fcbd4c4c26 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:15.242647 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc84132-42b2-4bbb-a048-a6fcbd4c4c26 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:15.243163 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc84132-42b2-4bbb-a048-a6fcbd4c4c26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:15.282755 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:15.308308 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-7bc84132-42b2-4bbb-a048-a6fcbd4c4c26 None None] MFA Rules not processed for user `9b5efa37b94b47949344d00ff88be4fb`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:15.314071 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-7bc84132-42b2-4bbb-a048-a6fcbd4c4c26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:15.494333 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1387/2773] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:15 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:15.500234 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2a2b5c-185f-4667-a037-859c6177e39d service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:15.502497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-6a2a2b5c-185f-4667-a037-859c6177e39d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:15.613179 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2a2b5c-185f-4667-a037-859c6177e39d service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:15.630134 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcc3620-3c13-4015-99e5-9982c2468148 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:15.630719 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcc3620-3c13-4015-99e5-9982c2468148 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:15.631005 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcc3620-3c13-4015-99e5-9982c2468148 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:15.632245 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcc3620-3c13-4015-99e5-9982c2468148 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:15.635095 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dcc3620-3c13-4015-99e5-9982c2468148 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:15.638860 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dcc3620-3c13-4015-99e5-9982c2468148 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:15.667113 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:15.773284 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1387/2774] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:14:15 2022] POST /identity/v3/projects => generated 385 bytes in 273 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:14:15.775441 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc84132-42b2-4bbb-a048-a6fcbd4c4c26 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:15.790134 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc84132-42b2-4bbb-a048-a6fcbd4c4c26 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:15.884531 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5c233a-cf2a-49ab-9cc8-25e1a7fd231a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:15.885507 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5c233a-cf2a-49ab-9cc8-25e1a7fd231a admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:15.885798 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5c233a-cf2a-49ab-9cc8-25e1a7fd231a admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:15.886077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5c233a-cf2a-49ab-9cc8-25e1a7fd231a admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:15.887010 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5c233a-cf2a-49ab-9cc8-25e1a7fd231a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:15.890942 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5c233a-cf2a-49ab-9cc8-25e1a7fd231a admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:16.057690 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1388/2775] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:14:15 2022] POST /identity/v3/users => generated 299 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:14:16.063383 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcc3620-3c13-4015-99e5-9982c2468148 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:16.080414 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcc3620-3c13-4015-99e5-9982c2468148 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:16.086873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a6747e-2f9d-49ec-9986-e9f3469aaace admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:16.087450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a6747e-2f9d-49ec-9986-e9f3469aaace admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:16.087790 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a6747e-2f9d-49ec-9986-e9f3469aaace admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:16.087986 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a6747e-2f9d-49ec-9986-e9f3469aaace admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:16.088644 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a6747e-2f9d-49ec-9986-e9f3469aaace admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:16.101790 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a6747e-2f9d-49ec-9986-e9f3469aaace admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:16.144293 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1388/2776] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:16 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:16.148935 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5c233a-cf2a-49ab-9cc8-25e1a7fd231a admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:16.190584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5c233a-cf2a-49ab-9cc8-25e1a7fd231a admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:16.199774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c18ae97-b627-44c6-9086-45a7dc1f93a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:16.201796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c18ae97-b627-44c6-9086-45a7dc1f93a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:16.202609 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c18ae97-b627-44c6-9086-45a7dc1f93a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:16.203048 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c18ae97-b627-44c6-9086-45a7dc1f93a1 admin admin] PATH_INFO: `/v3/projects/abca446946bf4b32a50b47d29e71cba0/users/00c145cda7e143299a1c2aff7d408fd7/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:16.227624 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c18ae97-b627-44c6-9086-45a7dc1f93a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=abca446946bf4b32a50b47d29e71cba0, user_id=00c145cda7e143299a1c2aff7d408fd7, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:16.230654 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c18ae97-b627-44c6-9086-45a7dc1f93a1 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:16.325380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1389/2777] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:14:16 2022] PUT /identity/v3/projects/abca446946bf4b32a50b47d29e71cba0/users/00c145cda7e143299a1c2aff7d408fd7/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:16.331230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a6747e-2f9d-49ec-9986-e9f3469aaace admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:16.356092 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a6747e-2f9d-49ec-9986-e9f3469aaace admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:16.432300 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6c380d-b3d2-4054-a6c7-806ad4ebeecb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:16.432918 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6c380d-b3d2-4054-a6c7-806ad4ebeecb admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:16.433236 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6c380d-b3d2-4054-a6c7-806ad4ebeecb admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:16.433515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6c380d-b3d2-4054-a6c7-806ad4ebeecb admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:16.434171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6c380d-b3d2-4054-a6c7-806ad4ebeecb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:16.436135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6c380d-b3d2-4054-a6c7-806ad4ebeecb admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:16.451190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1389/2778] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:16 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:16.452183 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c18ae97-b627-44c6-9086-45a7dc1f93a1 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:16.470502 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c18ae97-b627-44c6-9086-45a7dc1f93a1 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:16.478894 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6c380d-b3d2-4054-a6c7-806ad4ebeecb admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:16.485066 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eae5f6-43bb-442d-9442-9757bca24e42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:16.485992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eae5f6-43bb-442d-9442-9757bca24e42 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:16.486438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eae5f6-43bb-442d-9442-9757bca24e42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:16.486925 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eae5f6-43bb-442d-9442-9757bca24e42 admin admin] PATH_INFO: `/v3/projects/abca446946bf4b32a50b47d29e71cba0/users/00c145cda7e143299a1c2aff7d408fd7/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:16.488032 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4a6c380d-b3d2-4054-a6c7-806ad4ebeecb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:16.489799 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1eae5f6-43bb-442d-9442-9757bca24e42 admin admin] RBAC: Authorizing `identity:create_grant(project_id=abca446946bf4b32a50b47d29e71cba0, user_id=00c145cda7e143299a1c2aff7d408fd7, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:16.492854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1eae5f6-43bb-442d-9442-9757bca24e42 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:16.504997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6c380d-b3d2-4054-a6c7-806ad4ebeecb admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:16.507835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1390/2779] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:14:16 2022] PUT /identity/v3/projects/abca446946bf4b32a50b47d29e71cba0/users/00c145cda7e143299a1c2aff7d408fd7/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:14:16.633766 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ee083e-b9c6-4d4d-ae4f-cb08bb679990 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:16.633766 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ee083e-b9c6-4d4d-ae4f-cb08bb679990 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:16.633766 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ee083e-b9c6-4d4d-ae4f-cb08bb679990 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:16.633766 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ee083e-b9c6-4d4d-ae4f-cb08bb679990 admin admin] PATH_INFO: `/v3/users/13182cef4cf8458ab34779ffd7514c6c` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:16.637300 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ee083e-b9c6-4d4d-ae4f-cb08bb679990 admin admin] RBAC: Authorizing `identity:delete_user(user_id=13182cef4cf8458ab34779ffd7514c6c)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:16.640026 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ee083e-b9c6-4d4d-ae4f-cb08bb679990 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:16.876815 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-22ee083e-b9c6-4d4d-ae4f-cb08bb679990 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '13182cef4cf8458ab34779ffd7514c6c'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:16.986171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-22ee083e-b9c6-4d4d-ae4f-cb08bb679990 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '13182cef4cf8458ab34779ffd7514c6c'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:17.022034 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-22ee083e-b9c6-4d4d-ae4f-cb08bb679990 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '13182cef4cf8458ab34779ffd7514c6c'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:17.022807 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-22ee083e-b9c6-4d4d-ae4f-cb08bb679990 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '13182cef4cf8458ab34779ffd7514c6c'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:17.039261 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:17.042139 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-22ee083e-b9c6-4d4d-ae4f-cb08bb679990 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '13182cef4cf8458ab34779ffd7514c6c'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:17.049501 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1390/2780] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:14:16 2022] DELETE /identity/v3/users/13182cef4cf8458ab34779ffd7514c6c => generated 0 bytes in 571 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:17.382213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eae5f6-43bb-442d-9442-9757bca24e42 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:17.384045 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-a1eae5f6-43bb-442d-9442-9757bca24e42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:17.412064 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eae5f6-43bb-442d-9442-9757bca24e42 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:17.480028 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:17.491123 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafc0483-4357-4714-8966-85a2fed12aed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:17.491796 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafc0483-4357-4714-8966-85a2fed12aed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:17.492267 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafc0483-4357-4714-8966-85a2fed12aed admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:17.492990 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafc0483-4357-4714-8966-85a2fed12aed admin admin] PATH_INFO: `/v3/projects/735896cd6b3b4c448dd94b3bf7577cb8` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:17.494547 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafc0483-4357-4714-8966-85a2fed12aed admin admin] RBAC: Authorizing `identity:delete_project(project_id=735896cd6b3b4c448dd94b3bf7577cb8)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:17.496537 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafc0483-4357-4714-8966-85a2fed12aed admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:17.617343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-cafc0483-4357-4714-8966-85a2fed12aed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '735896cd6b3b4c448dd94b3bf7577cb8', 'request_id': 'req-cafc0483-4357-4714-8966-85a2fed12aed'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:17.624656 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-cafc0483-4357-4714-8966-85a2fed12aed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '735896cd6b3b4c448dd94b3bf7577cb8', 'request_id': 'req-cafc0483-4357-4714-8966-85a2fed12aed'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:17.707234 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-cafc0483-4357-4714-8966-85a2fed12aed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '735896cd6b3b4c448dd94b3bf7577cb8', 'request_id': 'req-cafc0483-4357-4714-8966-85a2fed12aed'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:17.721059 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-cafc0483-4357-4714-8966-85a2fed12aed admin admin] The token cache is being invalidate because project 735896cd6b3b4c448dd94b3bf7577cb8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:14:17.721490 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-cafc0483-4357-4714-8966-85a2fed12aed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:17.722157 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-cafc0483-4357-4714-8966-85a2fed12aed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:17.723678 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1391/2781] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:14:17 2022] DELETE /identity/v3/projects/735896cd6b3b4c448dd94b3bf7577cb8 => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:17.727915 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ee083e-b9c6-4d4d-ae4f-cb08bb679990 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:17.738343 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ee083e-b9c6-4d4d-ae4f-cb08bb679990 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:17.801185 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b69f27-b4c2-433f-ae67-f1e3e6f71897 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:17.801833 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b69f27-b4c2-433f-ae67-f1e3e6f71897 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:17.809319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b69f27-b4c2-433f-ae67-f1e3e6f71897 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:17.809319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b69f27-b4c2-433f-ae67-f1e3e6f71897 admin admin] PATH_INFO: `/v3/users/4306e3c30d854718a031bee8020cc83d` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:17.809319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b69f27-b4c2-433f-ae67-f1e3e6f71897 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4306e3c30d854718a031bee8020cc83d)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:17.809319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b69f27-b4c2-433f-ae67-f1e3e6f71897 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:18.021443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-78b69f27-b4c2-433f-ae67-f1e3e6f71897 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4306e3c30d854718a031bee8020cc83d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:18.092087 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-78b69f27-b4c2-433f-ae67-f1e3e6f71897 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4306e3c30d854718a031bee8020cc83d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:18.119171 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-78b69f27-b4c2-433f-ae67-f1e3e6f71897 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4306e3c30d854718a031bee8020cc83d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:18.120045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-78b69f27-b4c2-433f-ae67-f1e3e6f71897 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4306e3c30d854718a031bee8020cc83d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:18.141298 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-78b69f27-b4c2-433f-ae67-f1e3e6f71897 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4306e3c30d854718a031bee8020cc83d'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:18.155713 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1391/2782] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:14:17 2022] DELETE /identity/v3/users/4306e3c30d854718a031bee8020cc83d => generated 0 bytes in 428 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:18.518934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafc0483-4357-4714-8966-85a2fed12aed admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:18.521562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-cafc0483-4357-4714-8966-85a2fed12aed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:18.536706 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafc0483-4357-4714-8966-85a2fed12aed admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:18.669374 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dd045f-d5b8-47d1-b22d-34f778fc99f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:18.670061 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dd045f-d5b8-47d1-b22d-34f778fc99f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:18.670501 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dd045f-d5b8-47d1-b22d-34f778fc99f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:18.670911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dd045f-d5b8-47d1-b22d-34f778fc99f4 admin admin] PATH_INFO: `/v3/projects/2e0ecc6513e54f9597cef651eabc4879` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:18.672168 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07dd045f-d5b8-47d1-b22d-34f778fc99f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e0ecc6513e54f9597cef651eabc4879)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:18.674317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07dd045f-d5b8-47d1-b22d-34f778fc99f4 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:18.817735 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-07dd045f-d5b8-47d1-b22d-34f778fc99f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e0ecc6513e54f9597cef651eabc4879', 'request_id': 'req-07dd045f-d5b8-47d1-b22d-34f778fc99f4'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:18.824722 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-07dd045f-d5b8-47d1-b22d-34f778fc99f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e0ecc6513e54f9597cef651eabc4879', 'request_id': 'req-07dd045f-d5b8-47d1-b22d-34f778fc99f4'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:18.910835 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-07dd045f-d5b8-47d1-b22d-34f778fc99f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e0ecc6513e54f9597cef651eabc4879', 'request_id': 'req-07dd045f-d5b8-47d1-b22d-34f778fc99f4'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:18.926396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-07dd045f-d5b8-47d1-b22d-34f778fc99f4 admin admin] The token cache is being invalidate because project 2e0ecc6513e54f9597cef651eabc4879 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:14:18.926955 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-07dd045f-d5b8-47d1-b22d-34f778fc99f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:18.928478 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-07dd045f-d5b8-47d1-b22d-34f778fc99f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:18.930487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1392/2783] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:14:18 2022] DELETE /identity/v3/projects/2e0ecc6513e54f9597cef651eabc4879 => generated 0 bytes in 412 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:21.904394 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b69f27-b4c2-433f-ae67-f1e3e6f71897 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:21.906000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-78b69f27-b4c2-433f-ae67-f1e3e6f71897 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:21.920315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b69f27-b4c2-433f-ae67-f1e3e6f71897 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:21.991644 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bafc7af-862f-437b-9bbc-f3f434f8cb07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:21.992287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bafc7af-862f-437b-9bbc-f3f434f8cb07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:21.992569 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bafc7af-862f-437b-9bbc-f3f434f8cb07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:21.992839 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bafc7af-862f-437b-9bbc-f3f434f8cb07 admin admin] PATH_INFO: `/v3/users/a8da12374eb8481598219b9f818b8901` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:21.994268 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bafc7af-862f-437b-9bbc-f3f434f8cb07 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a8da12374eb8481598219b9f818b8901)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:21.996186 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bafc7af-862f-437b-9bbc-f3f434f8cb07 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:22.013931 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:22.254475 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-1bafc7af-862f-437b-9bbc-f3f434f8cb07 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a8da12374eb8481598219b9f818b8901'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:22.369382 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-1bafc7af-862f-437b-9bbc-f3f434f8cb07 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a8da12374eb8481598219b9f818b8901'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:22.383642 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-1bafc7af-862f-437b-9bbc-f3f434f8cb07 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a8da12374eb8481598219b9f818b8901'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:22.384416 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-1bafc7af-862f-437b-9bbc-f3f434f8cb07 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a8da12374eb8481598219b9f818b8901'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:22.392139 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-1bafc7af-862f-437b-9bbc-f3f434f8cb07 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a8da12374eb8481598219b9f818b8901'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:22.398712 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1392/2784] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:14:21 2022] DELETE /identity/v3/users/a8da12374eb8481598219b9f818b8901 => generated 0 bytes in 495 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:22.403895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955ab8d3-2371-4282-973d-45478e1b6abb None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:22.404417 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955ab8d3-2371-4282-973d-45478e1b6abb None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:22.404733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955ab8d3-2371-4282-973d-45478e1b6abb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:22.428345 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-955ab8d3-2371-4282-973d-45478e1b6abb None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:22.430235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-955ab8d3-2371-4282-973d-45478e1b6abb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:22.441923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:22.516823 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1393/2785] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:14:22.526494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bafc7af-862f-437b-9bbc-f3f434f8cb07 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:22.527974 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-1bafc7af-862f-437b-9bbc-f3f434f8cb07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:22.620903 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bafc7af-862f-437b-9bbc-f3f434f8cb07 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:22.625726 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e5b5b6-b320-4d85-b360-52d057a53220 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:22.626357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e5b5b6-b320-4d85-b360-52d057a53220 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:22.626636 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e5b5b6-b320-4d85-b360-52d057a53220 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:22.626908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e5b5b6-b320-4d85-b360-52d057a53220 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:22.639477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e5b5b6-b320-4d85-b360-52d057a53220 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:22.641265 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e5b5b6-b320-4d85-b360-52d057a53220 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:22.656334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1393/2786] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:22.947026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903f0d3e-a15f-456a-a653-c7c51b1e44fc None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:22.947277 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903f0d3e-a15f-456a-a653-c7c51b1e44fc None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:22.947562 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903f0d3e-a15f-456a-a653-c7c51b1e44fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:23.017751 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-903f0d3e-a15f-456a-a653-c7c51b1e44fc None None] MFA Rules not processed for user `00c145cda7e143299a1c2aff7d408fd7`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:23.019201 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-903f0d3e-a15f-456a-a653-c7c51b1e44fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:23.128392 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1394/2787] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:22 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:23.144160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e5b5b6-b320-4d85-b360-52d057a53220 service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:23.145618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a0e5b5b6-b320-4d85-b360-52d057a53220 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:23.221820 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e5b5b6-b320-4d85-b360-52d057a53220 service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:23.226321 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5b0a79-bf7e-4fde-8eb1-87cd123650f5 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '69153344fc4b45588bdbcecdfcfd7c3f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:23.227173 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5b0a79-bf7e-4fde-8eb1-87cd123650f5 service glance] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:23.230809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5b0a79-bf7e-4fde-8eb1-87cd123650f5 service glance] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:23.231275 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5b0a79-bf7e-4fde-8eb1-87cd123650f5 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:23.232375 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903f0d3e-a15f-456a-a653-c7c51b1e44fc None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:23.233960 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-1f5b0a79-bf7e-4fde-8eb1-87cd123650f5 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:23.236946 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-903f0d3e-a15f-456a-a653-c7c51b1e44fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:23.254353 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903f0d3e-a15f-456a-a653-c7c51b1e44fc None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:23.262357 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8737015f-a667-4ae7-81be-661d90bf9c83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:23.263075 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8737015f-a667-4ae7-81be-661d90bf9c83 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:23.263419 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8737015f-a667-4ae7-81be-661d90bf9c83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:23.264079 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8737015f-a667-4ae7-81be-661d90bf9c83 admin admin] PATH_INFO: `/v3/projects/45844a6d610a4549ab7b4a200a111a9f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:23.265825 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8737015f-a667-4ae7-81be-661d90bf9c83 admin admin] RBAC: Authorizing `identity:delete_project(project_id=45844a6d610a4549ab7b4a200a111a9f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:23.268729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8737015f-a667-4ae7-81be-661d90bf9c83 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:23.308673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:23.347564 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5b0a79-bf7e-4fde-8eb1-87cd123650f5 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:23.349713 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5b0a79-bf7e-4fde-8eb1-87cd123650f5 service glance] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:23.362052 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1394/2788] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:23 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:23.471291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8737015f-a667-4ae7-81be-661d90bf9c83 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '45844a6d610a4549ab7b4a200a111a9f', 'request_id': 'req-8737015f-a667-4ae7-81be-661d90bf9c83'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:23.477261 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8737015f-a667-4ae7-81be-661d90bf9c83 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '45844a6d610a4549ab7b4a200a111a9f', 'request_id': 'req-8737015f-a667-4ae7-81be-661d90bf9c83'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:23.573760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8737015f-a667-4ae7-81be-661d90bf9c83 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '45844a6d610a4549ab7b4a200a111a9f', 'request_id': 'req-8737015f-a667-4ae7-81be-661d90bf9c83'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:23.588280 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8737015f-a667-4ae7-81be-661d90bf9c83 admin admin] The token cache is being invalidate because project 45844a6d610a4549ab7b4a200a111a9f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:14:23.588569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8737015f-a667-4ae7-81be-661d90bf9c83 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:23.589389 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-8737015f-a667-4ae7-81be-661d90bf9c83 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:23.591291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1395/2789] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:14:23 2022] DELETE /identity/v3/projects/45844a6d610a4549ab7b4a200a111a9f => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:14:23.596413 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ea56a7-3b6e-4440-becf-45fb487dc528 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:23.596699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ea56a7-3b6e-4440-becf-45fb487dc528 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:23.596972 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ea56a7-3b6e-4440-becf-45fb487dc528 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:23.620216 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-94ea56a7-3b6e-4440-becf-45fb487dc528 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:23.621518 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-94ea56a7-3b6e-4440-becf-45fb487dc528 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:23.687150 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60127ca4-68c7-426f-b145-d4b4eb1a4788 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:23.687724 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60127ca4-68c7-426f-b145-d4b4eb1a4788 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:23.688136 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60127ca4-68c7-426f-b145-d4b4eb1a4788 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:23.706092 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1395/2790] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:23.713797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed251902-daf6-4ff2-9217-a1b6a3cc6866 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:23.714086 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed251902-daf6-4ff2-9217-a1b6a3cc6866 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:23.714372 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed251902-daf6-4ff2-9217-a1b6a3cc6866 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:23.730184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-60127ca4-68c7-426f-b145-d4b4eb1a4788 None None] MFA Rules not processed for user `70cef7c271e046a3b49230057cc2d28f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:23.732278 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-60127ca4-68c7-426f-b145-d4b4eb1a4788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:23.735380 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-ed251902-daf6-4ff2-9217-a1b6a3cc6866 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:23.736586 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ed251902-daf6-4ff2-9217-a1b6a3cc6866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:23.742416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:23.745716 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1396/2791] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:14:23.750950 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed251902-daf6-4ff2-9217-a1b6a3cc6866 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:23.763658 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed251902-daf6-4ff2-9217-a1b6a3cc6866 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:23.768864 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c8675a-32b6-4174-a8cf-da2e1dcb5afc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:23.769836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c8675a-32b6-4174-a8cf-da2e1dcb5afc admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:23.769981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c8675a-32b6-4174-a8cf-da2e1dcb5afc admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:23.770229 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c8675a-32b6-4174-a8cf-da2e1dcb5afc admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:23.770881 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c8675a-32b6-4174-a8cf-da2e1dcb5afc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:14:23.771187 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c8675a-32b6-4174-a8cf-da2e1dcb5afc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:23.773337 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c8675a-32b6-4174-a8cf-da2e1dcb5afc admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:23.789039 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1397/2792] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:14:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 04 01:14:23.790136 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c8675a-32b6-4174-a8cf-da2e1dcb5afc admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:23.802840 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c8675a-32b6-4174-a8cf-da2e1dcb5afc admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:23.808014 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26f5e3e-a292-46e4-a7ec-831fb03048f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:23.808619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26f5e3e-a292-46e4-a7ec-831fb03048f2 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:23.809490 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26f5e3e-a292-46e4-a7ec-831fb03048f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:23.809490 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26f5e3e-a292-46e4-a7ec-831fb03048f2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:23.810116 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26f5e3e-a292-46e4-a7ec-831fb03048f2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:23.812035 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26f5e3e-a292-46e4-a7ec-831fb03048f2 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:23.812576 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1396/2793] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 4 01:14:23 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:14:23.930790 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1398/2794] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:14:23 2022] POST /identity/v3/projects => generated 379 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:14:23.933315 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60127ca4-68c7-426f-b145-d4b4eb1a4788 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:23.949251 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60127ca4-68c7-426f-b145-d4b4eb1a4788 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:23.962439 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26f5e3e-a292-46e4-a7ec-831fb03048f2 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:23.964302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d26f5e3e-a292-46e4-a7ec-831fb03048f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:24.018923 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f825b5-e6e4-40bb-99ba-14c3b18265c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:24.019545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f825b5-e6e4-40bb-99ba-14c3b18265c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:24.019827 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f825b5-e6e4-40bb-99ba-14c3b18265c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:24.020104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f825b5-e6e4-40bb-99ba-14c3b18265c5 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:24.020760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f825b5-e6e4-40bb-99ba-14c3b18265c5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:24.022778 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f825b5-e6e4-40bb-99ba-14c3b18265c5 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:24.050690 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26f5e3e-a292-46e4-a7ec-831fb03048f2 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:24.057170 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0de6b86-4e44-4320-8f19-fc17b1726c39 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '123ab6ff796c4e599ce1d6f4095bd898', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:24.057778 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0de6b86-4e44-4320-8f19-fc17b1726c39 service swift] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:24.058135 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0de6b86-4e44-4320-8f19-fc17b1726c39 service swift] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:24.058464 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0de6b86-4e44-4320-8f19-fc17b1726c39 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:24.112946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:24.115472 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1397/2795] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:14:23 2022] POST /identity/v3/users => generated 290 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:14:24.116352 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f825b5-e6e4-40bb-99ba-14c3b18265c5 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:24.122536 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0de6b86-4e44-4320-8f19-fc17b1726c39 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:24.124783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0de6b86-4e44-4320-8f19-fc17b1726c39 service swift] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:24.126436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f825b5-e6e4-40bb-99ba-14c3b18265c5 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:24.131532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128db060-daf1-4243-8278-ebfae5803aa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:24.132496 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128db060-daf1-4243-8278-ebfae5803aa5 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:24.133157 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128db060-daf1-4243-8278-ebfae5803aa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:24.133582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128db060-daf1-4243-8278-ebfae5803aa5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:24.134258 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1399/2796] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 4 01:14:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 04 01:14:24.141505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128db060-daf1-4243-8278-ebfae5803aa5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:24.143121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128db060-daf1-4243-8278-ebfae5803aa5 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:24.160896 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1398/2797] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:24 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:24.170794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0de6b86-4e44-4320-8f19-fc17b1726c39 service swift] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:24.189062 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0de6b86-4e44-4320-8f19-fc17b1726c39 service swift] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:24.196379 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8339f799-cd26-484e-9174-c8078d4119dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:24.197434 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8339f799-cd26-484e-9174-c8078d4119dc admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:24.197904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8339f799-cd26-484e-9174-c8078d4119dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:24.206143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8339f799-cd26-484e-9174-c8078d4119dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:24.206143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8339f799-cd26-484e-9174-c8078d4119dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:24.206143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8339f799-cd26-484e-9174-c8078d4119dc admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:24.227507 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1400/2798] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:24 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:24.229068 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128db060-daf1-4243-8278-ebfae5803aa5 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:24.255425 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128db060-daf1-4243-8278-ebfae5803aa5 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:24.261491 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d95f44-0266-402e-a6d3-f7a3ee7b1560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:24.262245 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d95f44-0266-402e-a6d3-f7a3ee7b1560 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:24.262591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d95f44-0266-402e-a6d3-f7a3ee7b1560 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:24.262926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d95f44-0266-402e-a6d3-f7a3ee7b1560 admin admin] PATH_INFO: `/v3/projects/2506b96ac13541e6a7a47c5aedca67e5/users/48c4f0c78a364c00a793666eeb5a520c/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:24.284432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d95f44-0266-402e-a6d3-f7a3ee7b1560 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2506b96ac13541e6a7a47c5aedca67e5, user_id=48c4f0c78a364c00a793666eeb5a520c, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:24.287232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d95f44-0266-402e-a6d3-f7a3ee7b1560 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:24.402181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1399/2799] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:14:24 2022] PUT /identity/v3/projects/2506b96ac13541e6a7a47c5aedca67e5/users/48c4f0c78a364c00a793666eeb5a520c/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:24.407126 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab9b881-aadf-4715-8272-645fe2a1af6e None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:24.407503 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab9b881-aadf-4715-8272-645fe2a1af6e None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:24.407900 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab9b881-aadf-4715-8272-645fe2a1af6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:24.437128 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-9ab9b881-aadf-4715-8272-645fe2a1af6e None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:24.438823 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-9ab9b881-aadf-4715-8272-645fe2a1af6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:24.540348 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1401/2800] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:24.548385 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d95f44-0266-402e-a6d3-f7a3ee7b1560 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:24.551034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-04d95f44-0266-402e-a6d3-f7a3ee7b1560 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:24.641984 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d95f44-0266-402e-a6d3-f7a3ee7b1560 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:24.646924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c770a018-ce49-4b6a-a571-db83289eb184 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:24.647791 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c770a018-ce49-4b6a-a571-db83289eb184 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:24.648232 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c770a018-ce49-4b6a-a571-db83289eb184 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:24.648664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c770a018-ce49-4b6a-a571-db83289eb184 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:24.660603 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c770a018-ce49-4b6a-a571-db83289eb184 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:24.662336 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c770a018-ce49-4b6a-a571-db83289eb184 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:24.682027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1400/2801] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:31.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7c0f53-aed0-44b4-8246-73804675e486 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:31.808640 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7c0f53-aed0-44b4-8246-73804675e486 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:31.809205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7c0f53-aed0-44b4-8246-73804675e486 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:31.843573 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-5e7c0f53-aed0-44b4-8246-73804675e486 None None] MFA Rules not processed for user `48c4f0c78a364c00a793666eeb5a520c`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:31.844868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-5e7c0f53-aed0-44b4-8246-73804675e486 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:31.937668 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1402/2802] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:31 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:31.938386 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c770a018-ce49-4b6a-a571-db83289eb184 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:31.953964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c770a018-ce49-4b6a-a571-db83289eb184 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:31.959584 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2330ddf9-7d17-48c9-af0a-6a8369d9d43d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:31.960306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2330ddf9-7d17-48c9-af0a-6a8369d9d43d admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:31.961436 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2330ddf9-7d17-48c9-af0a-6a8369d9d43d admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:31.962055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2330ddf9-7d17-48c9-af0a-6a8369d9d43d admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:31.963453 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2330ddf9-7d17-48c9-af0a-6a8369d9d43d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:31.968802 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2330ddf9-7d17-48c9-af0a-6a8369d9d43d admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:31.993148 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:32.089244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1401/2803] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:14:31 2022] POST /identity/v3/projects => generated 379 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:14:32.093530 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7c0f53-aed0-44b4-8246-73804675e486 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:32.108830 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7c0f53-aed0-44b4-8246-73804675e486 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:32.181881 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b6f485-952d-4abe-a118-cd0afb2d4153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:32.182502 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b6f485-952d-4abe-a118-cd0afb2d4153 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:32.184935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b6f485-952d-4abe-a118-cd0afb2d4153 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:32.184935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b6f485-952d-4abe-a118-cd0afb2d4153 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:32.184935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b6f485-952d-4abe-a118-cd0afb2d4153 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:32.187453 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b6f485-952d-4abe-a118-cd0afb2d4153 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:32.273958 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:32.286617 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1403/2804] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:14:32 2022] POST /identity/v3/users => generated 296 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:14:32.291952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2330ddf9-7d17-48c9-af0a-6a8369d9d43d admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:32.308305 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2330ddf9-7d17-48c9-af0a-6a8369d9d43d admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:32.316617 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376c9989-5569-44b1-922f-742d78f1ffa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:32.317411 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376c9989-5569-44b1-922f-742d78f1ffa0 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:32.317738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376c9989-5569-44b1-922f-742d78f1ffa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:32.318018 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376c9989-5569-44b1-922f-742d78f1ffa0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:32.318760 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376c9989-5569-44b1-922f-742d78f1ffa0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:32.320491 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376c9989-5569-44b1-922f-742d78f1ffa0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:32.338882 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1402/2805] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:32 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:32.343173 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b6f485-952d-4abe-a118-cd0afb2d4153 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:32.354719 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b6f485-952d-4abe-a118-cd0afb2d4153 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:32.360461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6cd7305-da88-45ea-b908-173387779943 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:32.362687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cd7305-da88-45ea-b908-173387779943 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:32.362687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cd7305-da88-45ea-b908-173387779943 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:32.362687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6cd7305-da88-45ea-b908-173387779943 admin admin] PATH_INFO: `/v3/projects/fa1ba3942811492fb9ef388575baa4e2/users/eaaaa268a2664111b8a25f368bb354fc/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:32.381500 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6cd7305-da88-45ea-b908-173387779943 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa1ba3942811492fb9ef388575baa4e2, user_id=eaaaa268a2664111b8a25f368bb354fc, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:32.384239 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6cd7305-da88-45ea-b908-173387779943 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:32.730047 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1404/2806] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:14:32 2022] PUT /identity/v3/projects/fa1ba3942811492fb9ef388575baa4e2/users/eaaaa268a2664111b8a25f368bb354fc/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:32.734619 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376c9989-5569-44b1-922f-742d78f1ffa0 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:32.748812 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376c9989-5569-44b1-922f-742d78f1ffa0 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:32.812744 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078b0ab3-83a9-4d7b-90c3-8b6cf43c0cfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:32.813728 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078b0ab3-83a9-4d7b-90c3-8b6cf43c0cfa admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:32.814012 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078b0ab3-83a9-4d7b-90c3-8b6cf43c0cfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:32.814283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078b0ab3-83a9-4d7b-90c3-8b6cf43c0cfa admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:32.814934 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078b0ab3-83a9-4d7b-90c3-8b6cf43c0cfa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:32.816543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078b0ab3-83a9-4d7b-90c3-8b6cf43c0cfa admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:32.827938 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1403/2807] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:32 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:32.838436 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6cd7305-da88-45ea-b908-173387779943 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:32.851308 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6cd7305-da88-45ea-b908-173387779943 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:32.852163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277ddbe6-2f76-4450-8aa6-29a82531fb55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:32.852420 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277ddbe6-2f76-4450-8aa6-29a82531fb55 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:32.852904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277ddbe6-2f76-4450-8aa6-29a82531fb55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:32.853268 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277ddbe6-2f76-4450-8aa6-29a82531fb55 admin admin] PATH_INFO: `/v3/projects/fa1ba3942811492fb9ef388575baa4e2/users/eaaaa268a2664111b8a25f368bb354fc/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:32.855326 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277ddbe6-2f76-4450-8aa6-29a82531fb55 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa1ba3942811492fb9ef388575baa4e2, user_id=eaaaa268a2664111b8a25f368bb354fc, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:32.858397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277ddbe6-2f76-4450-8aa6-29a82531fb55 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:32.877018 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1405/2808] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:14:32 2022] PUT /identity/v3/projects/fa1ba3942811492fb9ef388575baa4e2/users/eaaaa268a2664111b8a25f368bb354fc/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:36.349739 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078b0ab3-83a9-4d7b-90c3-8b6cf43c0cfa admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:36.422056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-078b0ab3-83a9-4d7b-90c3-8b6cf43c0cfa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:36.445893 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078b0ab3-83a9-4d7b-90c3-8b6cf43c0cfa admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:36.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc78d60-d922-4cb7-a791-9469310a3523 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:36.548467 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc78d60-d922-4cb7-a791-9469310a3523 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:36.549114 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc78d60-d922-4cb7-a791-9469310a3523 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:36.549733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc78d60-d922-4cb7-a791-9469310a3523 admin admin] PATH_INFO: `/v3/users/1bfe064e18d542d1a8b587c9ba76e83c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:36.551164 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc78d60-d922-4cb7-a791-9469310a3523 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1bfe064e18d542d1a8b587c9ba76e83c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:36.553416 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc78d60-d922-4cb7-a791-9469310a3523 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:37.071345 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7cc78d60-d922-4cb7-a791-9469310a3523 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1bfe064e18d542d1a8b587c9ba76e83c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:37.072110 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7cc78d60-d922-4cb7-a791-9469310a3523 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1bfe064e18d542d1a8b587c9ba76e83c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:37.191963 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7cc78d60-d922-4cb7-a791-9469310a3523 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1bfe064e18d542d1a8b587c9ba76e83c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:37.208058 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7cc78d60-d922-4cb7-a791-9469310a3523 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1bfe064e18d542d1a8b587c9ba76e83c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:37.209065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-7cc78d60-d922-4cb7-a791-9469310a3523 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1bfe064e18d542d1a8b587c9ba76e83c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:37.219367 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:INSERT=2,keystone:SELECT=88,keystone:DELETE=10 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:37.231719 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1404/2809] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:14:36 2022] DELETE /identity/v3/users/1bfe064e18d542d1a8b587c9ba76e83c => generated 0 bytes in 882 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:14:37.236458 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7654abef-def9-42df-8ba2-bdeee8dc5e79 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:37.236982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7654abef-def9-42df-8ba2-bdeee8dc5e79 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:37.236982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7654abef-def9-42df-8ba2-bdeee8dc5e79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:37.260506 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-7654abef-def9-42df-8ba2-bdeee8dc5e79 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:37.263742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7654abef-def9-42df-8ba2-bdeee8dc5e79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:37.354027 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1406/2810] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:37.363794 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc78d60-d922-4cb7-a791-9469310a3523 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:37.365317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-7cc78d60-d922-4cb7-a791-9469310a3523 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:37.477315 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc78d60-d922-4cb7-a791-9469310a3523 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:37.481866 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e5f916-e15f-4f44-a7f9-aa6054f46e47 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:37.482821 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e5f916-e15f-4f44-a7f9-aa6054f46e47 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:37.483772 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e5f916-e15f-4f44-a7f9-aa6054f46e47 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:37.484203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e5f916-e15f-4f44-a7f9-aa6054f46e47 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:37.499067 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e5f916-e15f-4f44-a7f9-aa6054f46e47 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:37.503380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e5f916-e15f-4f44-a7f9-aa6054f46e47 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:37.517917 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1405/2811] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:38.263122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7654abef-def9-42df-8ba2-bdeee8dc5e79 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:38.264568 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7654abef-def9-42df-8ba2-bdeee8dc5e79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:38.278457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7654abef-def9-42df-8ba2-bdeee8dc5e79 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:38.285947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf623dc6-d022-40a9-b336-889ac6f2184e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:38.286888 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf623dc6-d022-40a9-b336-889ac6f2184e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:38.287338 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf623dc6-d022-40a9-b336-889ac6f2184e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:38.296548 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf623dc6-d022-40a9-b336-889ac6f2184e admin admin] PATH_INFO: `/v3/projects/b5bfefa807364899ae84d6ae0acf6c89` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:38.299873 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf623dc6-d022-40a9-b336-889ac6f2184e admin admin] RBAC: Authorizing `identity:delete_project(project_id=b5bfefa807364899ae84d6ae0acf6c89)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:38.302030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf623dc6-d022-40a9-b336-889ac6f2184e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:38.413487 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cf623dc6-d022-40a9-b336-889ac6f2184e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b5bfefa807364899ae84d6ae0acf6c89', 'request_id': 'req-cf623dc6-d022-40a9-b336-889ac6f2184e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:38.422762 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cf623dc6-d022-40a9-b336-889ac6f2184e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b5bfefa807364899ae84d6ae0acf6c89', 'request_id': 'req-cf623dc6-d022-40a9-b336-889ac6f2184e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:38.433885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cf623dc6-d022-40a9-b336-889ac6f2184e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b5bfefa807364899ae84d6ae0acf6c89', 'request_id': 'req-cf623dc6-d022-40a9-b336-889ac6f2184e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:38.502051 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cf623dc6-d022-40a9-b336-889ac6f2184e admin admin] The token cache is being invalidate because project b5bfefa807364899ae84d6ae0acf6c89 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:14:38.502288 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cf623dc6-d022-40a9-b336-889ac6f2184e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:38.502894 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cf623dc6-d022-40a9-b336-889ac6f2184e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:38.504927 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1407/2812] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:14:38 2022] DELETE /identity/v3/projects/b5bfefa807364899ae84d6ae0acf6c89 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:38.508578 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e5f916-e15f-4f44-a7f9-aa6054f46e47 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:38.520156 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e5f916-e15f-4f44-a7f9-aa6054f46e47 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:38.655482 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11193e24-1d99-49ad-b3c3-12b688eea515 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:38.656802 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11193e24-1d99-49ad-b3c3-12b688eea515 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:38.657218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11193e24-1d99-49ad-b3c3-12b688eea515 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:38.657535 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11193e24-1d99-49ad-b3c3-12b688eea515 admin admin] PATH_INFO: `/v3/users/c50d8f2f03584a7994b238f0dc6d75e8` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:38.659176 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11193e24-1d99-49ad-b3c3-12b688eea515 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c50d8f2f03584a7994b238f0dc6d75e8)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:38.661615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11193e24-1d99-49ad-b3c3-12b688eea515 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:38.718650 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0236437-899b-4b08-b287-ced732f4b293 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:38.718935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0236437-899b-4b08-b287-ced732f4b293 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:38.719204 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0236437-899b-4b08-b287-ced732f4b293 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:38.743030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-d0236437-899b-4b08-b287-ced732f4b293 None None] MFA Rules not processed for user `9b5efa37b94b47949344d00ff88be4fb`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:38.746093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d0236437-899b-4b08-b287-ced732f4b293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:38.802696 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:38.819285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1408/2813] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:38 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:38.898941 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-11193e24-1d99-49ad-b3c3-12b688eea515 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c50d8f2f03584a7994b238f0dc6d75e8'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:38.901390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-11193e24-1d99-49ad-b3c3-12b688eea515 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c50d8f2f03584a7994b238f0dc6d75e8'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:39.042177 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-11193e24-1d99-49ad-b3c3-12b688eea515 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c50d8f2f03584a7994b238f0dc6d75e8'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:39.059777 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-11193e24-1d99-49ad-b3c3-12b688eea515 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c50d8f2f03584a7994b238f0dc6d75e8'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:39.060691 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-11193e24-1d99-49ad-b3c3-12b688eea515 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c50d8f2f03584a7994b238f0dc6d75e8'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:39.076252 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1406/2814] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:14:38 2022] DELETE /identity/v3/users/c50d8f2f03584a7994b238f0dc6d75e8 => generated 0 bytes in 568 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:14:39.216197 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0236437-899b-4b08-b287-ced732f4b293 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:39.217713 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d0236437-899b-4b08-b287-ced732f4b293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:39.299842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0236437-899b-4b08-b287-ced732f4b293 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:39.409389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec20615c-5ede-4a82-8b11-5f0d2c8c9f7d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:39.410041 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec20615c-5ede-4a82-8b11-5f0d2c8c9f7d service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:39.410312 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec20615c-5ede-4a82-8b11-5f0d2c8c9f7d service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:39.410588 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec20615c-5ede-4a82-8b11-5f0d2c8c9f7d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:39.412456 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ec20615c-5ede-4a82-8b11-5f0d2c8c9f7d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:39.436570 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec20615c-5ede-4a82-8b11-5f0d2c8c9f7d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:39.438079 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec20615c-5ede-4a82-8b11-5f0d2c8c9f7d service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:39.452826 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1409/2815] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:39 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:39.718491 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11193e24-1d99-49ad-b3c3-12b688eea515 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:39.720789 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-11193e24-1d99-49ad-b3c3-12b688eea515 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:39.766064 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:39.851026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11193e24-1d99-49ad-b3c3-12b688eea515 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:39.866776 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66babfc-0d1e-48a4-b6e0-47e0319bf3ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:39.867442 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66babfc-0d1e-48a4-b6e0-47e0319bf3ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:39.867759 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66babfc-0d1e-48a4-b6e0-47e0319bf3ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:39.868033 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66babfc-0d1e-48a4-b6e0-47e0319bf3ec admin admin] PATH_INFO: `/v3/projects/136968e58b1b42289f63983c8d7f4927` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:39.869599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66babfc-0d1e-48a4-b6e0-47e0319bf3ec admin admin] RBAC: Authorizing `identity:delete_project(project_id=136968e58b1b42289f63983c8d7f4927)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:39.871942 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66babfc-0d1e-48a4-b6e0-47e0319bf3ec admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:40.041783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e66babfc-0d1e-48a4-b6e0-47e0319bf3ec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '136968e58b1b42289f63983c8d7f4927', 'request_id': 'req-e66babfc-0d1e-48a4-b6e0-47e0319bf3ec'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:40.041783 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e66babfc-0d1e-48a4-b6e0-47e0319bf3ec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '136968e58b1b42289f63983c8d7f4927', 'request_id': 'req-e66babfc-0d1e-48a4-b6e0-47e0319bf3ec'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:40.117450 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e66babfc-0d1e-48a4-b6e0-47e0319bf3ec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '136968e58b1b42289f63983c8d7f4927', 'request_id': 'req-e66babfc-0d1e-48a4-b6e0-47e0319bf3ec'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:40.131467 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e66babfc-0d1e-48a4-b6e0-47e0319bf3ec admin admin] The token cache is being invalidate because project 136968e58b1b42289f63983c8d7f4927 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:14:40.131757 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e66babfc-0d1e-48a4-b6e0-47e0319bf3ec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:40.132526 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-e66babfc-0d1e-48a4-b6e0-47e0319bf3ec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:40.137774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1407/2816] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:14:39 2022] DELETE /identity/v3/projects/136968e58b1b42289f63983c8d7f4927 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:40.140667 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f5a4d8-4195-4fe5-8d68-175879aaed78 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:40.141176 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f5a4d8-4195-4fe5-8d68-175879aaed78 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:40.142043 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f5a4d8-4195-4fe5-8d68-175879aaed78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:40.171659 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-67f5a4d8-4195-4fe5-8d68-175879aaed78 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:40.173187 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-67f5a4d8-4195-4fe5-8d68-175879aaed78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:40.260546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1410/2817] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:40.270252 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef58a18-94fc-4032-aaa4-dd9fececdebe None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:40.270591 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef58a18-94fc-4032-aaa4-dd9fececdebe None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:40.275187 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef58a18-94fc-4032-aaa4-dd9fececdebe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:40.296759 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-cef58a18-94fc-4032-aaa4-dd9fececdebe None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:40.299311 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-cef58a18-94fc-4032-aaa4-dd9fececdebe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:40.319176 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1408/2818] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:40.319348 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f5a4d8-4195-4fe5-8d68-175879aaed78 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:40.333172 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f5a4d8-4195-4fe5-8d68-175879aaed78 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:40.338714 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff31599-8b68-4d99-92c6-8cdc1a6d9e12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:40.342097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff31599-8b68-4d99-92c6-8cdc1a6d9e12 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:40.342097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff31599-8b68-4d99-92c6-8cdc1a6d9e12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:40.342097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff31599-8b68-4d99-92c6-8cdc1a6d9e12 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:40.342097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff31599-8b68-4d99-92c6-8cdc1a6d9e12 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:14:40.342783 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff31599-8b68-4d99-92c6-8cdc1a6d9e12 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:40.344693 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff31599-8b68-4d99-92c6-8cdc1a6d9e12 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:40.364474 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1411/2819] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:14:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:14:40.365326 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef58a18-94fc-4032-aaa4-dd9fececdebe None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:40.377174 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef58a18-94fc-4032-aaa4-dd9fececdebe None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:40.384536 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b715b4-5252-43b6-804c-7918cf79bdb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:40.385203 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b715b4-5252-43b6-804c-7918cf79bdb1 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:40.385488 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b715b4-5252-43b6-804c-7918cf79bdb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:40.389115 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b715b4-5252-43b6-804c-7918cf79bdb1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:40.390071 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b715b4-5252-43b6-804c-7918cf79bdb1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:40.392774 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b715b4-5252-43b6-804c-7918cf79bdb1 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:40.396654 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb989e6-75f1-487a-a262-4033037cffa3 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:40.397023 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb989e6-75f1-487a-a262-4033037cffa3 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:40.397406 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb989e6-75f1-487a-a262-4033037cffa3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:40.435930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-ecb989e6-75f1-487a-a262-4033037cffa3 None None] MFA Rules not processed for user `eaaaa268a2664111b8a25f368bb354fc`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:40.437579 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ecb989e6-75f1-487a-a262-4033037cffa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:40.471170 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:40.490016 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1409/2820] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:14:40 2022] POST /identity/v3/projects => generated 361 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:14:40.491077 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b715b4-5252-43b6-804c-7918cf79bdb1 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:40.528136 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1412/2821] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:40 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:40.542240 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb989e6-75f1-487a-a262-4033037cffa3 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:40.543675 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ecb989e6-75f1-487a-a262-4033037cffa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:40.569853 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b715b4-5252-43b6-804c-7918cf79bdb1 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:40.574701 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52428208-f446-43f3-992f-29451588ad17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:40.575357 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52428208-f446-43f3-992f-29451588ad17 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:40.575638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52428208-f446-43f3-992f-29451588ad17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:40.575912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52428208-f446-43f3-992f-29451588ad17 admin admin] PATH_INFO: `/v3/users` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:40.576552 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52428208-f446-43f3-992f-29451588ad17 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:40.578525 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52428208-f446-43f3-992f-29451588ad17 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:40.655935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb989e6-75f1-487a-a262-4033037cffa3 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:40.705181 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1410/2822] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:14:40 2022] POST /identity/v3/users => generated 281 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:14:40.710483 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52428208-f446-43f3-992f-29451588ad17 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:40.725372 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c1c6a1-2e54-4477-9277-3dbcfa7b946c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:40.726786 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52428208-f446-43f3-992f-29451588ad17 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:40.727238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c1c6a1-2e54-4477-9277-3dbcfa7b946c service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:40.727238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c1c6a1-2e54-4477-9277-3dbcfa7b946c service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:40.727726 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c1c6a1-2e54-4477-9277-3dbcfa7b946c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:40.729832 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d2c1c6a1-2e54-4477-9277-3dbcfa7b946c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:40.733004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995b0e2e-bb5d-4ce1-87d3-63005236e501 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:40.733752 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995b0e2e-bb5d-4ce1-87d3-63005236e501 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:40.734034 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995b0e2e-bb5d-4ce1-87d3-63005236e501 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:40.734313 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995b0e2e-bb5d-4ce1-87d3-63005236e501 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:40.734977 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995b0e2e-bb5d-4ce1-87d3-63005236e501 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:40.736585 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995b0e2e-bb5d-4ce1-87d3-63005236e501 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:40.742867 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:40.752182 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1411/2823] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:40 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:40.755546 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995b0e2e-bb5d-4ce1-87d3-63005236e501 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:40.764673 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995b0e2e-bb5d-4ce1-87d3-63005236e501 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:40.770555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf63cc7-7a16-403a-b52b-c4153a73d80b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:40.771167 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf63cc7-7a16-403a-b52b-c4153a73d80b admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:40.771448 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf63cc7-7a16-403a-b52b-c4153a73d80b admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:40.771717 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf63cc7-7a16-403a-b52b-c4153a73d80b admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:40.772429 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf63cc7-7a16-403a-b52b-c4153a73d80b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:40.774615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf63cc7-7a16-403a-b52b-c4153a73d80b admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:40.791570 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1412/2824] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:40 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:40.796009 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf63cc7-7a16-403a-b52b-c4153a73d80b admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:40.799266 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c1c6a1-2e54-4477-9277-3dbcfa7b946c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:40.800895 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c1c6a1-2e54-4477-9277-3dbcfa7b946c service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:40.804739 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:40.805926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf63cc7-7a16-403a-b52b-c4153a73d80b admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:40.810852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1413/2825] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:40 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 269 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:40.810992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba0bf57-87db-4f5a-b753-b127213c345a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:40.811495 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba0bf57-87db-4f5a-b753-b127213c345a admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:40.811776 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba0bf57-87db-4f5a-b753-b127213c345a admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:40.825515 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba0bf57-87db-4f5a-b753-b127213c345a admin admin] PATH_INFO: `/v3/projects/2ed7dbfcd5804556b351d0113810c6b5/users/ae92228cd82147aaa9d2e19a747f8d0f/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:40.827959 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba0bf57-87db-4f5a-b753-b127213c345a admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ed7dbfcd5804556b351d0113810c6b5, user_id=ae92228cd82147aaa9d2e19a747f8d0f, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:40.830912 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba0bf57-87db-4f5a-b753-b127213c345a admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:40.882956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1413/2826] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:14:40 2022] PUT /identity/v3/projects/2ed7dbfcd5804556b351d0113810c6b5/users/ae92228cd82147aaa9d2e19a747f8d0f/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:40.894072 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0697aa4f-7b15-4f1b-972b-5027879e9677 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:40.894358 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0697aa4f-7b15-4f1b-972b-5027879e9677 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:40.894633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0697aa4f-7b15-4f1b-972b-5027879e9677 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:40.929136 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-0697aa4f-7b15-4f1b-972b-5027879e9677 None None] MFA Rules not processed for user `ae92228cd82147aaa9d2e19a747f8d0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:40.930430 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-0697aa4f-7b15-4f1b-972b-5027879e9677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:41.019334 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba0bf57-87db-4f5a-b753-b127213c345a admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:41.021302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1414/2827] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:40 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:41.114928 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba0bf57-87db-4f5a-b753-b127213c345a admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:41.120355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3189f6cd-fb36-45d8-ac32-c14b844938cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:41.120954 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3189f6cd-fb36-45d8-ac32-c14b844938cd admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:41.121317 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3189f6cd-fb36-45d8-ac32-c14b844938cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:41.126398 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3189f6cd-fb36-45d8-ac32-c14b844938cd admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:41.126398 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3189f6cd-fb36-45d8-ac32-c14b844938cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:41.126398 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3189f6cd-fb36-45d8-ac32-c14b844938cd admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:41.251898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1414/2828] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:14:41 2022] POST /identity/v3/projects => generated 365 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:14:41.255916 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0697aa4f-7b15-4f1b-972b-5027879e9677 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:41.267386 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0697aa4f-7b15-4f1b-972b-5027879e9677 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:41.326429 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223dc6c4-efc2-4828-9f91-a4e360255237 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:41.327141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223dc6c4-efc2-4828-9f91-a4e360255237 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:41.327302 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223dc6c4-efc2-4828-9f91-a4e360255237 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:41.327578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223dc6c4-efc2-4828-9f91-a4e360255237 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:41.328217 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223dc6c4-efc2-4828-9f91-a4e360255237 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:41.330192 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223dc6c4-efc2-4828-9f91-a4e360255237 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:41.431516 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1415/2829] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:14:41 2022] POST /identity/v3/users => generated 289 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:14:41.436264 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3189f6cd-fb36-45d8-ac32-c14b844938cd admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:41.446792 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3189f6cd-fb36-45d8-ac32-c14b844938cd admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:41.451872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a89d0e-9b42-494c-8645-d07cd988c4ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:41.452516 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a89d0e-9b42-494c-8645-d07cd988c4ce admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:41.452733 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a89d0e-9b42-494c-8645-d07cd988c4ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:41.453035 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a89d0e-9b42-494c-8645-d07cd988c4ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:41.453710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a89d0e-9b42-494c-8645-d07cd988c4ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:41.455687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a89d0e-9b42-494c-8645-d07cd988c4ce admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:41.471961 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1415/2830] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:41 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:41.476443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223dc6c4-efc2-4828-9f91-a4e360255237 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:41.487930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223dc6c4-efc2-4828-9f91-a4e360255237 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:41.493055 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6419a0-fe64-4d77-8fb7-1775ec051345 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:41.493847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6419a0-fe64-4d77-8fb7-1775ec051345 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:41.494394 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6419a0-fe64-4d77-8fb7-1775ec051345 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:41.494671 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6419a0-fe64-4d77-8fb7-1775ec051345 admin admin] PATH_INFO: `/v3/projects/3641208e829b41528f461d9bf8d61c5a/users/9801f211d2a44ea591e449a49e320277/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:41.511048 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6419a0-fe64-4d77-8fb7-1775ec051345 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3641208e829b41528f461d9bf8d61c5a, user_id=9801f211d2a44ea591e449a49e320277, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:41.515963 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6419a0-fe64-4d77-8fb7-1775ec051345 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:41.610241 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1416/2831] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:14:41 2022] PUT /identity/v3/projects/3641208e829b41528f461d9bf8d61c5a/users/9801f211d2a44ea591e449a49e320277/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:41.610450 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a89d0e-9b42-494c-8645-d07cd988c4ce admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:41.615916 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a89d0e-9b42-494c-8645-d07cd988c4ce admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:41.707729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b859333-0dfa-4de0-83ce-00045683da6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:41.708338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b859333-0dfa-4de0-83ce-00045683da6b admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:41.708613 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b859333-0dfa-4de0-83ce-00045683da6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:41.708915 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b859333-0dfa-4de0-83ce-00045683da6b admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:41.709603 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b859333-0dfa-4de0-83ce-00045683da6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:41.711175 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b859333-0dfa-4de0-83ce-00045683da6b admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:41.731806 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1416/2832] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:41 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:41.732365 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6419a0-fe64-4d77-8fb7-1775ec051345 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:41.740831 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6419a0-fe64-4d77-8fb7-1775ec051345 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:41.746063 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b750bef-058b-4f5d-a715-5f36a59684ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:41.748271 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b750bef-058b-4f5d-a715-5f36a59684ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:41.748271 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b750bef-058b-4f5d-a715-5f36a59684ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:41.748486 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b750bef-058b-4f5d-a715-5f36a59684ae admin admin] PATH_INFO: `/v3/projects/3641208e829b41528f461d9bf8d61c5a/users/9801f211d2a44ea591e449a49e320277/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:41.750397 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b750bef-058b-4f5d-a715-5f36a59684ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=3641208e829b41528f461d9bf8d61c5a, user_id=9801f211d2a44ea591e449a49e320277, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:41.753055 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b750bef-058b-4f5d-a715-5f36a59684ae admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:41.764836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1417/2833] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:14:41 2022] PUT /identity/v3/projects/3641208e829b41528f461d9bf8d61c5a/users/9801f211d2a44ea591e449a49e320277/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:41.776056 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f50c05-dbf4-4390-a07d-8ee8f5db5703 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:41.776343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f50c05-dbf4-4390-a07d-8ee8f5db5703 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:41.776619 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f50c05-dbf4-4390-a07d-8ee8f5db5703 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:41.805105 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-b7f50c05-dbf4-4390-a07d-8ee8f5db5703 None None] MFA Rules not processed for user `9801f211d2a44ea591e449a49e320277`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:41.812932 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b7f50c05-dbf4-4390-a07d-8ee8f5db5703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:41.833275 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:41.884467 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1417/2834] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:41 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:41.897442 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b750bef-058b-4f5d-a715-5f36a59684ae admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:41.899642 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4b750bef-058b-4f5d-a715-5f36a59684ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:41.948489 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:41.963005 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b750bef-058b-4f5d-a715-5f36a59684ae admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:41.966776 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c871ff33-ce3c-451a-a93b-372d3ea9bf87 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:41.967372 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c871ff33-ce3c-451a-a93b-372d3ea9bf87 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:41.967677 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c871ff33-ce3c-451a-a93b-372d3ea9bf87 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:41.967946 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c871ff33-ce3c-451a-a93b-372d3ea9bf87 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:41.976914 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c871ff33-ce3c-451a-a93b-372d3ea9bf87 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:41.978384 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c871ff33-ce3c-451a-a93b-372d3ea9bf87 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:41.988085 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1418/2835] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:41 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:42.253528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f50c05-dbf4-4390-a07d-8ee8f5db5703 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:42.266638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f50c05-dbf4-4390-a07d-8ee8f5db5703 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:42.271017 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14200a5f-d46e-43c2-a644-6542ba39bc7d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:42.271615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14200a5f-d46e-43c2-a644-6542ba39bc7d service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:42.272632 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14200a5f-d46e-43c2-a644-6542ba39bc7d service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:42.272632 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14200a5f-d46e-43c2-a644-6542ba39bc7d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:42.331734 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14200a5f-d46e-43c2-a644-6542ba39bc7d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:42.333227 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14200a5f-d46e-43c2-a644-6542ba39bc7d service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:42.343252 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1418/2836] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:42 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:44.338037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976af4f6-93f4-40b0-8b39-3c40919de514 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:44.338037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976af4f6-93f4-40b0-8b39-3c40919de514 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:44.338037 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976af4f6-93f4-40b0-8b39-3c40919de514 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:44.361520 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-976af4f6-93f4-40b0-8b39-3c40919de514 None None] MFA Rules not processed for user `70cef7c271e046a3b49230057cc2d28f`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:44.362749 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-976af4f6-93f4-40b0-8b39-3c40919de514 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:44.495669 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1419/2837] 10.222.0.42 () {58 vars in 1009 bytes} [Sun Sep 4 01:14:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:44.512704 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14200a5f-d46e-43c2-a644-6542ba39bc7d service cinder] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:44.512704 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-14200a5f-d46e-43c2-a644-6542ba39bc7d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:44.611581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14200a5f-d46e-43c2-a644-6542ba39bc7d service cinder] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:44.624238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5990517a-64c8-4650-acbd-4faf967f9020 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '123ab6ff796c4e599ce1d6f4095bd898', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:44.625015 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5990517a-64c8-4650-acbd-4faf967f9020 service swift] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:44.625409 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5990517a-64c8-4650-acbd-4faf967f9020 service swift] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:44.625805 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5990517a-64c8-4650-acbd-4faf967f9020 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:44.646244 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5990517a-64c8-4650-acbd-4faf967f9020 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:44.649746 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5990517a-64c8-4650-acbd-4faf967f9020 service swift] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:44.663098 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1419/2838] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 4 01:14:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 04 01:14:46.298441 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976af4f6-93f4-40b0-8b39-3c40919de514 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:46.393641 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976af4f6-93f4-40b0-8b39-3c40919de514 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:46.399533 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dfadfb-5d7e-4e96-835e-509372675eea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:46.400548 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dfadfb-5d7e-4e96-835e-509372675eea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:46.401055 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dfadfb-5d7e-4e96-835e-509372675eea admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:46.401600 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dfadfb-5d7e-4e96-835e-509372675eea admin admin] PATH_INFO: `/v3/users/ae92228cd82147aaa9d2e19a747f8d0f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:46.403195 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1dfadfb-5d7e-4e96-835e-509372675eea admin admin] RBAC: Authorizing `identity:delete_user(user_id=ae92228cd82147aaa9d2e19a747f8d0f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:46.405653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1dfadfb-5d7e-4e96-835e-509372675eea admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:46.567678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-a1dfadfb-5d7e-4e96-835e-509372675eea admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ae92228cd82147aaa9d2e19a747f8d0f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:46.626095 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-a1dfadfb-5d7e-4e96-835e-509372675eea admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ae92228cd82147aaa9d2e19a747f8d0f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:46.678274 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-a1dfadfb-5d7e-4e96-835e-509372675eea admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ae92228cd82147aaa9d2e19a747f8d0f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:46.679360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-a1dfadfb-5d7e-4e96-835e-509372675eea admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ae92228cd82147aaa9d2e19a747f8d0f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:46.706426 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-a1dfadfb-5d7e-4e96-835e-509372675eea admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ae92228cd82147aaa9d2e19a747f8d0f'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:46.714744 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1420/2839] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:14:46 2022] DELETE /identity/v3/users/ae92228cd82147aaa9d2e19a747f8d0f => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:46.719372 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03a31c9-7d5b-4c0b-9fc2-67ea2c4284f1 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:46.719893 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03a31c9-7d5b-4c0b-9fc2-67ea2c4284f1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:46.719983 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03a31c9-7d5b-4c0b-9fc2-67ea2c4284f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:46.741135 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-d03a31c9-7d5b-4c0b-9fc2-67ea2c4284f1 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:46.748548 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d03a31c9-7d5b-4c0b-9fc2-67ea2c4284f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:46.847759 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:46.908717 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1420/2840] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:46.927000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dfadfb-5d7e-4e96-835e-509372675eea admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:46.929248 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a1dfadfb-5d7e-4e96-835e-509372675eea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:46.966470 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:47.042567 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dfadfb-5d7e-4e96-835e-509372675eea admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:47.051166 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b60027-46f0-44f4-b967-2bb5257c5f72 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:47.055285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b60027-46f0-44f4-b967-2bb5257c5f72 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:47.055285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b60027-46f0-44f4-b967-2bb5257c5f72 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:47.055285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b60027-46f0-44f4-b967-2bb5257c5f72 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:47.065605 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b60027-46f0-44f4-b967-2bb5257c5f72 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:47.067678 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b60027-46f0-44f4-b967-2bb5257c5f72 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:47.081594 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1421/2841] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:47.593590 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03a31c9-7d5b-4c0b-9fc2-67ea2c4284f1 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:47.595178 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d03a31c9-7d5b-4c0b-9fc2-67ea2c4284f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:47.611599 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03a31c9-7d5b-4c0b-9fc2-67ea2c4284f1 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:47.616701 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0062255b-32aa-4043-94da-19a3fcd6d67b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:47.617451 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0062255b-32aa-4043-94da-19a3fcd6d67b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:47.617798 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0062255b-32aa-4043-94da-19a3fcd6d67b admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:47.618503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0062255b-32aa-4043-94da-19a3fcd6d67b admin admin] PATH_INFO: `/v3/projects/2ed7dbfcd5804556b351d0113810c6b5` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:47.619647 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0062255b-32aa-4043-94da-19a3fcd6d67b admin admin] RBAC: Authorizing `identity:delete_project(project_id=2ed7dbfcd5804556b351d0113810c6b5)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:47.622484 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0062255b-32aa-4043-94da-19a3fcd6d67b admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:47.752597 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0062255b-32aa-4043-94da-19a3fcd6d67b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2ed7dbfcd5804556b351d0113810c6b5', 'request_id': 'req-0062255b-32aa-4043-94da-19a3fcd6d67b'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:47.759505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0062255b-32aa-4043-94da-19a3fcd6d67b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2ed7dbfcd5804556b351d0113810c6b5', 'request_id': 'req-0062255b-32aa-4043-94da-19a3fcd6d67b'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:47.818685 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0062255b-32aa-4043-94da-19a3fcd6d67b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2ed7dbfcd5804556b351d0113810c6b5', 'request_id': 'req-0062255b-32aa-4043-94da-19a3fcd6d67b'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:47.837111 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0062255b-32aa-4043-94da-19a3fcd6d67b admin admin] The token cache is being invalidate because project 2ed7dbfcd5804556b351d0113810c6b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:14:47.837577 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0062255b-32aa-4043-94da-19a3fcd6d67b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:47.838337 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0062255b-32aa-4043-94da-19a3fcd6d67b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:47.840742 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1421/2842] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:14:47 2022] DELETE /identity/v3/projects/2ed7dbfcd5804556b351d0113810c6b5 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:47.844757 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b60027-46f0-44f4-b967-2bb5257c5f72 service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:47.855130 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b60027-46f0-44f4-b967-2bb5257c5f72 service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:47.954698 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f852384b-fcf0-40b4-b0f9-64af648d1193 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:47.955537 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f852384b-fcf0-40b4-b0f9-64af648d1193 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:47.955896 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f852384b-fcf0-40b4-b0f9-64af648d1193 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:47.956285 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f852384b-fcf0-40b4-b0f9-64af648d1193 admin admin] PATH_INFO: `/v3/users/9801f211d2a44ea591e449a49e320277` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:47.957809 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f852384b-fcf0-40b4-b0f9-64af648d1193 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9801f211d2a44ea591e449a49e320277)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:47.960031 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f852384b-fcf0-40b4-b0f9-64af648d1193 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:48.196874 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f852384b-fcf0-40b4-b0f9-64af648d1193 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9801f211d2a44ea591e449a49e320277'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:48.294327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f852384b-fcf0-40b4-b0f9-64af648d1193 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9801f211d2a44ea591e449a49e320277'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:48.313442 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f852384b-fcf0-40b4-b0f9-64af648d1193 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9801f211d2a44ea591e449a49e320277'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:48.314438 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f852384b-fcf0-40b4-b0f9-64af648d1193 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9801f211d2a44ea591e449a49e320277'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:48.322462 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-f852384b-fcf0-40b4-b0f9-64af648d1193 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9801f211d2a44ea591e449a49e320277'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:48.330060 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1422/2843] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:14:47 2022] DELETE /identity/v3/users/9801f211d2a44ea591e449a49e320277 => generated 0 bytes in 486 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:48.856128 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0062255b-32aa-4043-94da-19a3fcd6d67b admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:48.857321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-0062255b-32aa-4043-94da-19a3fcd6d67b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:48.873427 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0062255b-32aa-4043-94da-19a3fcd6d67b admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:48.963162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a88cb7-108b-43f3-9fdd-e7acf750eb44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:48.964089 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a88cb7-108b-43f3-9fdd-e7acf750eb44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:48.965007 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a88cb7-108b-43f3-9fdd-e7acf750eb44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:48.965510 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a88cb7-108b-43f3-9fdd-e7acf750eb44 admin admin] PATH_INFO: `/v3/projects/3641208e829b41528f461d9bf8d61c5a` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:48.967487 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a88cb7-108b-43f3-9fdd-e7acf750eb44 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3641208e829b41528f461d9bf8d61c5a)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:48.969470 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a88cb7-108b-43f3-9fdd-e7acf750eb44 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:49.078432 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-03a88cb7-108b-43f3-9fdd-e7acf750eb44 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3641208e829b41528f461d9bf8d61c5a', 'request_id': 'req-03a88cb7-108b-43f3-9fdd-e7acf750eb44'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:49.084397 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-03a88cb7-108b-43f3-9fdd-e7acf750eb44 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3641208e829b41528f461d9bf8d61c5a', 'request_id': 'req-03a88cb7-108b-43f3-9fdd-e7acf750eb44'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:49.169055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-03a88cb7-108b-43f3-9fdd-e7acf750eb44 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3641208e829b41528f461d9bf8d61c5a', 'request_id': 'req-03a88cb7-108b-43f3-9fdd-e7acf750eb44'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:49.182380 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-03a88cb7-108b-43f3-9fdd-e7acf750eb44 admin admin] The token cache is being invalidate because project 3641208e829b41528f461d9bf8d61c5a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:14:49.182677 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-03a88cb7-108b-43f3-9fdd-e7acf750eb44 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:49.183346 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-03a88cb7-108b-43f3-9fdd-e7acf750eb44 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:14:49.185209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1422/2844] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:14:48 2022] DELETE /identity/v3/projects/3641208e829b41528f461d9bf8d61c5a => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:49.190443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3edd84c-6fdf-4137-bd56-32c715b8877f None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:49.197966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3edd84c-6fdf-4137-bd56-32c715b8877f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:49.197966 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3edd84c-6fdf-4137-bd56-32c715b8877f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:49.212976 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-a3edd84c-6fdf-4137-bd56-32c715b8877f None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:49.214389 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-a3edd84c-6fdf-4137-bd56-32c715b8877f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:49.250530 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:49.291867 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1423/2845] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:49.300505 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e604a5-e01e-4910-abf6-8215010a5330 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:49.300794 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e604a5-e01e-4910-abf6-8215010a5330 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:49.301102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e604a5-e01e-4910-abf6-8215010a5330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:49.314284 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:49.327614 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-37e604a5-e01e-4910-abf6-8215010a5330 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:49.328847 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-37e604a5-e01e-4910-abf6-8215010a5330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:49.340643 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1423/2846] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:49.344127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3edd84c-6fdf-4137-bd56-32c715b8877f None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:49.354802 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3edd84c-6fdf-4137-bd56-32c715b8877f None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:49.364190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61200f7-0855-4462-b0da-c69483a1f902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:49.364190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61200f7-0855-4462-b0da-c69483a1f902 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:49.364190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61200f7-0855-4462-b0da-c69483a1f902 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:49.364190 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61200f7-0855-4462-b0da-c69483a1f902 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:49.364707 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61200f7-0855-4462-b0da-c69483a1f902 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:14:49.364707 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61200f7-0855-4462-b0da-c69483a1f902 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:49.364916 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61200f7-0855-4462-b0da-c69483a1f902 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:49.392804 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e604a5-e01e-4910-abf6-8215010a5330 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:49.398788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1424/2847] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:14:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:14:49.403714 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e604a5-e01e-4910-abf6-8215010a5330 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:49.410299 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8cc1c4-4cf5-4a4f-aae2-f4c03819bd7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:49.444162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8cc1c4-4cf5-4a4f-aae2-f4c03819bd7a admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:49.444162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8cc1c4-4cf5-4a4f-aae2-f4c03819bd7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:49.444162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8cc1c4-4cf5-4a4f-aae2-f4c03819bd7a admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:49.444162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8cc1c4-4cf5-4a4f-aae2-f4c03819bd7a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:49.444162 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8cc1c4-4cf5-4a4f-aae2-f4c03819bd7a admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:49.751612 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1424/2848] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:14:49 2022] POST /identity/v3/projects => generated 373 bytes in 358 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:14:49.754881 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61200f7-0855-4462-b0da-c69483a1f902 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:49.766127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61200f7-0855-4462-b0da-c69483a1f902 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:49.840773 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204ee76-ea45-42b2-98f3-c507e389a805 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:49.841637 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204ee76-ea45-42b2-98f3-c507e389a805 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:49.842172 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204ee76-ea45-42b2-98f3-c507e389a805 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:49.842517 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204ee76-ea45-42b2-98f3-c507e389a805 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:49.843233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d204ee76-ea45-42b2-98f3-c507e389a805 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:49.845239 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d204ee76-ea45-42b2-98f3-c507e389a805 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:49.946114 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1425/2849] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:14:49 2022] POST /identity/v3/users => generated 287 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:14:49.960550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8cc1c4-4cf5-4a4f-aae2-f4c03819bd7a admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:49.978188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8cc1c4-4cf5-4a4f-aae2-f4c03819bd7a admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:49.978769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237583c1-f6ed-4b51-af99-85259d371115 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:49.979407 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237583c1-f6ed-4b51-af99-85259d371115 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:49.979687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237583c1-f6ed-4b51-af99-85259d371115 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:49.979962 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237583c1-f6ed-4b51-af99-85259d371115 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:49.980623 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237583c1-f6ed-4b51-af99-85259d371115 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:49.982313 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237583c1-f6ed-4b51-af99-85259d371115 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:49.997463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1425/2850] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:49 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:50.000462 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204ee76-ea45-42b2-98f3-c507e389a805 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:50.014075 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204ee76-ea45-42b2-98f3-c507e389a805 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:50.021345 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6abfbf-2a7b-4c5b-8560-529eeac868e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:50.022357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6abfbf-2a7b-4c5b-8560-529eeac868e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:50.022802 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6abfbf-2a7b-4c5b-8560-529eeac868e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:50.023254 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6abfbf-2a7b-4c5b-8560-529eeac868e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:50.024139 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6abfbf-2a7b-4c5b-8560-529eeac868e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:50.025999 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6abfbf-2a7b-4c5b-8560-529eeac868e8 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:50.045194 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1426/2851] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:49 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:50.046025 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237583c1-f6ed-4b51-af99-85259d371115 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:50.059658 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237583c1-f6ed-4b51-af99-85259d371115 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:50.064869 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ce5b00-1f8c-45ec-9ce3-4b434a555a3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:50.065528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ce5b00-1f8c-45ec-9ce3-4b434a555a3e admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:50.065949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ce5b00-1f8c-45ec-9ce3-4b434a555a3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:50.066231 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ce5b00-1f8c-45ec-9ce3-4b434a555a3e admin admin] PATH_INFO: `/v3/projects/59ba1ddb45594bbd9c53c2ec9f5a8c80/users/7cadd7fb96b14688afbdcbe1e8fae759/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:50.085264 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ce5b00-1f8c-45ec-9ce3-4b434a555a3e admin admin] RBAC: Authorizing `identity:create_grant(project_id=59ba1ddb45594bbd9c53c2ec9f5a8c80, user_id=7cadd7fb96b14688afbdcbe1e8fae759, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:50.088325 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ce5b00-1f8c-45ec-9ce3-4b434a555a3e admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:50.186390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1426/2852] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:14:50 2022] PUT /identity/v3/projects/59ba1ddb45594bbd9c53c2ec9f5a8c80/users/7cadd7fb96b14688afbdcbe1e8fae759/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:50.188119 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7cc0bf-0682-4718-b6e3-8718354fa6c0 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:50.188427 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7cc0bf-0682-4718-b6e3-8718354fa6c0 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:50.188682 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7cc0bf-0682-4718-b6e3-8718354fa6c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:50.208808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-1b7cc0bf-0682-4718-b6e3-8718354fa6c0 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:50.210055 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-1b7cc0bf-0682-4718-b6e3-8718354fa6c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:50.296141 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1427/2853] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:14:50.305962 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ce5b00-1f8c-45ec-9ce3-4b434a555a3e admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:50.307829 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-93ce5b00-1f8c-45ec-9ce3-4b434a555a3e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:50.380080 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ce5b00-1f8c-45ec-9ce3-4b434a555a3e admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:50.383754 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8e4073-c54d-49cb-99c3-2484c32bcea8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:50.384351 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8e4073-c54d-49cb-99c3-2484c32bcea8 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:50.384629 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8e4073-c54d-49cb-99c3-2484c32bcea8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:50.384896 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8e4073-c54d-49cb-99c3-2484c32bcea8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:50.397820 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8e4073-c54d-49cb-99c3-2484c32bcea8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:50.399274 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8e4073-c54d-49cb-99c3-2484c32bcea8 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:50.410477 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1427/2854] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:51.748212 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e666da3-3b3c-4e69-9352-896e5a141f27 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:51.753217 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e666da3-3b3c-4e69-9352-896e5a141f27 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:51.753217 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e666da3-3b3c-4e69-9352-896e5a141f27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:51.785116 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-1e666da3-3b3c-4e69-9352-896e5a141f27 None None] MFA Rules not processed for user `7cadd7fb96b14688afbdcbe1e8fae759`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:51.786357 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-1e666da3-3b3c-4e69-9352-896e5a141f27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:51.819700 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:51.885884 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1428/2855] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:51 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:51.888809 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8e4073-c54d-49cb-99c3-2484c32bcea8 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:51.899871 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8e4073-c54d-49cb-99c3-2484c32bcea8 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:51.909331 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1576b0f-b8d0-455d-9a45-3a07b342de14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:51.910165 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1576b0f-b8d0-455d-9a45-3a07b342de14 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:51.911655 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1576b0f-b8d0-455d-9a45-3a07b342de14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:51.912343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1576b0f-b8d0-455d-9a45-3a07b342de14 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:51.913201 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1576b0f-b8d0-455d-9a45-3a07b342de14 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:51.921301 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1576b0f-b8d0-455d-9a45-3a07b342de14 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:52.050944 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e666da3-3b3c-4e69-9352-896e5a141f27 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:52.053025 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1428/2856] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:14:51 2022] POST /identity/v3/projects => generated 373 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:14:52.072657 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e666da3-3b3c-4e69-9352-896e5a141f27 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:52.153387 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa81d00-08ff-4a94-8d6d-e81ca8bc0825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:52.159342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa81d00-08ff-4a94-8d6d-e81ca8bc0825 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:52.159342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa81d00-08ff-4a94-8d6d-e81ca8bc0825 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:52.159342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa81d00-08ff-4a94-8d6d-e81ca8bc0825 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:52.159342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa81d00-08ff-4a94-8d6d-e81ca8bc0825 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:52.159342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa81d00-08ff-4a94-8d6d-e81ca8bc0825 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:52.295613 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1429/2857] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:14:52 2022] POST /identity/v3/users => generated 293 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:14:52.299338 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1576b0f-b8d0-455d-9a45-3a07b342de14 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:52.310468 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1576b0f-b8d0-455d-9a45-3a07b342de14 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:52.319116 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13102d6b-7fe0-4b72-b02f-67f46dac6186 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:52.319754 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13102d6b-7fe0-4b72-b02f-67f46dac6186 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:52.320033 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13102d6b-7fe0-4b72-b02f-67f46dac6186 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:52.320310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13102d6b-7fe0-4b72-b02f-67f46dac6186 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:52.321026 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13102d6b-7fe0-4b72-b02f-67f46dac6186 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:52.322696 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13102d6b-7fe0-4b72-b02f-67f46dac6186 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:52.338947 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1429/2858] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:52 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:14:52.343377 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa81d00-08ff-4a94-8d6d-e81ca8bc0825 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:52.358419 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa81d00-08ff-4a94-8d6d-e81ca8bc0825 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:52.363967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b734dce-0711-4984-8d94-b92140e48947 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:52.364633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b734dce-0711-4984-8d94-b92140e48947 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:52.365010 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b734dce-0711-4984-8d94-b92140e48947 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:52.370453 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b734dce-0711-4984-8d94-b92140e48947 admin admin] PATH_INFO: `/v3/projects/8d01b0cc21194a16b3dbf2cf7e390ca8/users/98d90e5ae1ff4b91939582ec99c561c6/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:52.386542 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b734dce-0711-4984-8d94-b92140e48947 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d01b0cc21194a16b3dbf2cf7e390ca8, user_id=98d90e5ae1ff4b91939582ec99c561c6, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:52.391942 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b734dce-0711-4984-8d94-b92140e48947 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:52.484266 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1430/2859] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:14:52 2022] PUT /identity/v3/projects/8d01b0cc21194a16b3dbf2cf7e390ca8/users/98d90e5ae1ff4b91939582ec99c561c6/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:14:52.488160 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13102d6b-7fe0-4b72-b02f-67f46dac6186 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:52.501708 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13102d6b-7fe0-4b72-b02f-67f46dac6186 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:52.608210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5935fdd9-24ea-42e6-a7e5-f9db779d5f3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:52.609094 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5935fdd9-24ea-42e6-a7e5-f9db779d5f3d admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:52.609564 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5935fdd9-24ea-42e6-a7e5-f9db779d5f3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:52.609990 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5935fdd9-24ea-42e6-a7e5-f9db779d5f3d admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:52.611042 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5935fdd9-24ea-42e6-a7e5-f9db779d5f3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:52.613543 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5935fdd9-24ea-42e6-a7e5-f9db779d5f3d admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:52.615988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:52.632793 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1430/2860] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:14:52 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:14:52.634061 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b734dce-0711-4984-8d94-b92140e48947 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:52.645814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b734dce-0711-4984-8d94-b92140e48947 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:52.650635 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79531075-7dfe-4d97-8f8e-9d6eb492db88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:52.651332 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79531075-7dfe-4d97-8f8e-9d6eb492db88 admin admin] REQUEST_METHOD: `PUT` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:52.651612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79531075-7dfe-4d97-8f8e-9d6eb492db88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:52.651885 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79531075-7dfe-4d97-8f8e-9d6eb492db88 admin admin] PATH_INFO: `/v3/projects/8d01b0cc21194a16b3dbf2cf7e390ca8/users/98d90e5ae1ff4b91939582ec99c561c6/roles/19c006412bec45dc8febd7cc86b9678f` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:52.653953 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79531075-7dfe-4d97-8f8e-9d6eb492db88 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d01b0cc21194a16b3dbf2cf7e390ca8, user_id=98d90e5ae1ff4b91939582ec99c561c6, role_id=19c006412bec45dc8febd7cc86b9678f)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:52.656862 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79531075-7dfe-4d97-8f8e-9d6eb492db88 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:52.668765 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1431/2861] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:14:52 2022] PUT /identity/v3/projects/8d01b0cc21194a16b3dbf2cf7e390ca8/users/98d90e5ae1ff4b91939582ec99c561c6/roles/19c006412bec45dc8febd7cc86b9678f => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:14:54.656414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8ec9cb-25ba-49c6-b7b8-f0d7a9e33557 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:54.667730 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8ec9cb-25ba-49c6-b7b8-f0d7a9e33557 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:54.667730 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8ec9cb-25ba-49c6-b7b8-f0d7a9e33557 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:54.694238 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-2c8ec9cb-25ba-49c6-b7b8-f0d7a9e33557 None None] MFA Rules not processed for user `98d90e5ae1ff4b91939582ec99c561c6`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:14:54.695514 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2c8ec9cb-25ba-49c6-b7b8-f0d7a9e33557 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:54.806310 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1431/2862] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:14:54 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:14:54.819283 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79531075-7dfe-4d97-8f8e-9d6eb492db88 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:54.821266 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-79531075-7dfe-4d97-8f8e-9d6eb492db88 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:54.887905 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79531075-7dfe-4d97-8f8e-9d6eb492db88 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:54.891624 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435d692d-07a6-477e-8220-9cc42e1fbf6e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:54.892359 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435d692d-07a6-477e-8220-9cc42e1fbf6e service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:54.892648 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435d692d-07a6-477e-8220-9cc42e1fbf6e service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:54.892904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435d692d-07a6-477e-8220-9cc42e1fbf6e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:54.928295 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:14:54.960289 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435d692d-07a6-477e-8220-9cc42e1fbf6e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:54.961837 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435d692d-07a6-477e-8220-9cc42e1fbf6e service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:54.971457 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1432/2863] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:54 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:55.697837 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8ec9cb-25ba-49c6-b7b8-f0d7a9e33557 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:55.699966 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2c8ec9cb-25ba-49c6-b7b8-f0d7a9e33557 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:55.758057 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8ec9cb-25ba-49c6-b7b8-f0d7a9e33557 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:55.761635 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9fde6d-0d4e-46ac-9887-32b2ed0c4c1a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:55.762297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9fde6d-0d4e-46ac-9887-32b2ed0c4c1a service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:55.762571 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9fde6d-0d4e-46ac-9887-32b2ed0c4c1a service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:55.762905 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9fde6d-0d4e-46ac-9887-32b2ed0c4c1a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:55.764710 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-cd9fde6d-0d4e-46ac-9887-32b2ed0c4c1a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:55.841321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9fde6d-0d4e-46ac-9887-32b2ed0c4c1a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:55.842762 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9fde6d-0d4e-46ac-9887-32b2ed0c4c1a service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:55.853659 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1432/2864] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:56.675567 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435d692d-07a6-477e-8220-9cc42e1fbf6e service cinder] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:56.684478 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435d692d-07a6-477e-8220-9cc42e1fbf6e service cinder] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:56.688018 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a5c346-c467-4348-ab2c-2c22c462a670 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:56.688606 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a5c346-c467-4348-ab2c-2c22c462a670 service cinder] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:56.688886 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a5c346-c467-4348-ab2c-2c22c462a670 service cinder] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:56.689196 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a5c346-c467-4348-ab2c-2c22c462a670 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:56.698007 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a5c346-c467-4348-ab2c-2c22c462a670 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:56.699938 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a5c346-c467-4348-ab2c-2c22c462a670 service cinder] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:56.709402 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1433/2865] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:56 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:14:57.029631 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9fde6d-0d4e-46ac-9887-32b2ed0c4c1a service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:14:57.038528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9fde6d-0d4e-46ac-9887-32b2ed0c4c1a service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:14:57.042304 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1714ad6a-a09e-4a54-be74-87da216da660 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:14:57.042931 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1714ad6a-a09e-4a54-be74-87da216da660 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:14:57.043281 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1714ad6a-a09e-4a54-be74-87da216da660 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:14:57.043581 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1714ad6a-a09e-4a54-be74-87da216da660 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:14:57.045449 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-1714ad6a-a09e-4a54-be74-87da216da660 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:14:57.059784 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1714ad6a-a09e-4a54-be74-87da216da660 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:14:57.061223 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1714ad6a-a09e-4a54-be74-87da216da660 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:14:57.072438 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1433/2866] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:14:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:15:00.772748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8101b0c3-f0ac-45c4-9a96-b7aace4221e6 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:00.773336 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8101b0c3-f0ac-45c4-9a96-b7aace4221e6 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:00.773336 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8101b0c3-f0ac-45c4-9a96-b7aace4221e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:00.793534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-8101b0c3-f0ac-45c4-9a96-b7aace4221e6 None None] MFA Rules not processed for user `48c4f0c78a364c00a793666eeb5a520c`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:15:00.794852 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-8101b0c3-f0ac-45c4-9a96-b7aace4221e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:00.856108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1434/2867] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:15:00 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:15:07.067186 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=96 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:15:10.366706 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1714ad6a-a09e-4a54-be74-87da216da660 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:10.379772 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1714ad6a-a09e-4a54-be74-87da216da660 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:10.384184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2508c61e-4957-411b-a2cd-af89b9339ec9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:10.384947 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2508c61e-4957-411b-a2cd-af89b9339ec9 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:10.385342 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2508c61e-4957-411b-a2cd-af89b9339ec9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:10.386656 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2508c61e-4957-411b-a2cd-af89b9339ec9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:10.389062 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-2508c61e-4957-411b-a2cd-af89b9339ec9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:10.478345 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2508c61e-4957-411b-a2cd-af89b9339ec9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:10.479898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2508c61e-4957-411b-a2cd-af89b9339ec9 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:10.492977 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1434/2868] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:15:10 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:15:10.848658 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=44 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:15:13.418334 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8101b0c3-f0ac-45c4-9a96-b7aace4221e6 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:13.522545 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8101b0c3-f0ac-45c4-9a96-b7aace4221e6 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:13.527791 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf21896e-1cda-4c9b-bdd3-6cb8ed3d8942 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:13.530926 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf21896e-1cda-4c9b-bdd3-6cb8ed3d8942 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:13.530926 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf21896e-1cda-4c9b-bdd3-6cb8ed3d8942 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:13.530926 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf21896e-1cda-4c9b-bdd3-6cb8ed3d8942 admin admin] PATH_INFO: `/v3/users/7cadd7fb96b14688afbdcbe1e8fae759` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:13.530926 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf21896e-1cda-4c9b-bdd3-6cb8ed3d8942 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7cadd7fb96b14688afbdcbe1e8fae759)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:13.533210 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf21896e-1cda-4c9b-bdd3-6cb8ed3d8942 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:13.747111 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cf21896e-1cda-4c9b-bdd3-6cb8ed3d8942 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7cadd7fb96b14688afbdcbe1e8fae759'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:13.837169 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cf21896e-1cda-4c9b-bdd3-6cb8ed3d8942 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7cadd7fb96b14688afbdcbe1e8fae759'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:13.854000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cf21896e-1cda-4c9b-bdd3-6cb8ed3d8942 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7cadd7fb96b14688afbdcbe1e8fae759'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:13.855531 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cf21896e-1cda-4c9b-bdd3-6cb8ed3d8942 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7cadd7fb96b14688afbdcbe1e8fae759'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:13.864680 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-cf21896e-1cda-4c9b-bdd3-6cb8ed3d8942 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7cadd7fb96b14688afbdcbe1e8fae759'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:13.871572 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1435/2869] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:15:13 2022] DELETE /identity/v3/users/7cadd7fb96b14688afbdcbe1e8fae759 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:15:14.113518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2508c61e-4957-411b-a2cd-af89b9339ec9 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:14.129439 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2508c61e-4957-411b-a2cd-af89b9339ec9 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:14.198002 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54fe317-5f58-472c-bb47-b10509fff55d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:14.200850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54fe317-5f58-472c-bb47-b10509fff55d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:14.200850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54fe317-5f58-472c-bb47-b10509fff55d admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:14.200850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54fe317-5f58-472c-bb47-b10509fff55d admin admin] PATH_INFO: `/v3/projects/59ba1ddb45594bbd9c53c2ec9f5a8c80` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:14.200850 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54fe317-5f58-472c-bb47-b10509fff55d admin admin] RBAC: Authorizing `identity:delete_project(project_id=59ba1ddb45594bbd9c53c2ec9f5a8c80)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:14.202551 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54fe317-5f58-472c-bb47-b10509fff55d admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:14.324957 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-f54fe317-5f58-472c-bb47-b10509fff55d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '59ba1ddb45594bbd9c53c2ec9f5a8c80', 'request_id': 'req-f54fe317-5f58-472c-bb47-b10509fff55d'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:14.331075 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-f54fe317-5f58-472c-bb47-b10509fff55d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '59ba1ddb45594bbd9c53c2ec9f5a8c80', 'request_id': 'req-f54fe317-5f58-472c-bb47-b10509fff55d'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:14.425513 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-f54fe317-5f58-472c-bb47-b10509fff55d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '59ba1ddb45594bbd9c53c2ec9f5a8c80', 'request_id': 'req-f54fe317-5f58-472c-bb47-b10509fff55d'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:14.439293 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-f54fe317-5f58-472c-bb47-b10509fff55d admin admin] The token cache is being invalidate because project 59ba1ddb45594bbd9c53c2ec9f5a8c80 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:15:14.439636 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-f54fe317-5f58-472c-bb47-b10509fff55d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:14.440308 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-f54fe317-5f58-472c-bb47-b10509fff55d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:14.441889 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1435/2870] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:15:14 2022] DELETE /identity/v3/projects/59ba1ddb45594bbd9c53c2ec9f5a8c80 => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:15:14.445909 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf21896e-1cda-4c9b-bdd3-6cb8ed3d8942 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:14.447346 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-cf21896e-1cda-4c9b-bdd3-6cb8ed3d8942 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:14.461578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf21896e-1cda-4c9b-bdd3-6cb8ed3d8942 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:14.521030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d22234e-7967-4e01-8bd1-2a8aef8dc223 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:14.521763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d22234e-7967-4e01-8bd1-2a8aef8dc223 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:14.522043 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d22234e-7967-4e01-8bd1-2a8aef8dc223 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:14.522314 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d22234e-7967-4e01-8bd1-2a8aef8dc223 admin admin] PATH_INFO: `/v3/users/98d90e5ae1ff4b91939582ec99c561c6` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:14.523448 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d22234e-7967-4e01-8bd1-2a8aef8dc223 admin admin] RBAC: Authorizing `identity:delete_user(user_id=98d90e5ae1ff4b91939582ec99c561c6)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:14.525495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d22234e-7967-4e01-8bd1-2a8aef8dc223 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:14.547731 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:15:14.698467 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-6d22234e-7967-4e01-8bd1-2a8aef8dc223 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '98d90e5ae1ff4b91939582ec99c561c6'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:14.790733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-6d22234e-7967-4e01-8bd1-2a8aef8dc223 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '98d90e5ae1ff4b91939582ec99c561c6'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:14.805353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-6d22234e-7967-4e01-8bd1-2a8aef8dc223 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '98d90e5ae1ff4b91939582ec99c561c6'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:14.806094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-6d22234e-7967-4e01-8bd1-2a8aef8dc223 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '98d90e5ae1ff4b91939582ec99c561c6'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:14.813994 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-6d22234e-7967-4e01-8bd1-2a8aef8dc223 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '98d90e5ae1ff4b91939582ec99c561c6'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:14.820347 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1436/2871] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:15:14 2022] DELETE /identity/v3/users/98d90e5ae1ff4b91939582ec99c561c6 => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:15:15.084030 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54fe317-5f58-472c-bb47-b10509fff55d admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:15.085474 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f54fe317-5f58-472c-bb47-b10509fff55d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:15.103406 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54fe317-5f58-472c-bb47-b10509fff55d admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:15.125959 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:15:15.162301 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acea47ee-b6c9-4a03-8fd1-999c01640a41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:15.162935 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acea47ee-b6c9-4a03-8fd1-999c01640a41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:15.163211 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acea47ee-b6c9-4a03-8fd1-999c01640a41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:15.163482 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acea47ee-b6c9-4a03-8fd1-999c01640a41 admin admin] PATH_INFO: `/v3/projects/8d01b0cc21194a16b3dbf2cf7e390ca8` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:15.164598 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acea47ee-b6c9-4a03-8fd1-999c01640a41 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d01b0cc21194a16b3dbf2cf7e390ca8)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:15.166518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acea47ee-b6c9-4a03-8fd1-999c01640a41 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:15.299694 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-acea47ee-b6c9-4a03-8fd1-999c01640a41 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d01b0cc21194a16b3dbf2cf7e390ca8', 'request_id': 'req-acea47ee-b6c9-4a03-8fd1-999c01640a41'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:15.305145 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-acea47ee-b6c9-4a03-8fd1-999c01640a41 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d01b0cc21194a16b3dbf2cf7e390ca8', 'request_id': 'req-acea47ee-b6c9-4a03-8fd1-999c01640a41'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:15.402430 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-acea47ee-b6c9-4a03-8fd1-999c01640a41 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d01b0cc21194a16b3dbf2cf7e390ca8', 'request_id': 'req-acea47ee-b6c9-4a03-8fd1-999c01640a41'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:15.414775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-acea47ee-b6c9-4a03-8fd1-999c01640a41 admin admin] The token cache is being invalidate because project 8d01b0cc21194a16b3dbf2cf7e390ca8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:15:15.415061 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-acea47ee-b6c9-4a03-8fd1-999c01640a41 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:15.415727 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-acea47ee-b6c9-4a03-8fd1-999c01640a41 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:15.417450 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1436/2872] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:15:15 2022] DELETE /identity/v3/projects/8d01b0cc21194a16b3dbf2cf7e390ca8 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:15:15.422361 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd4a5be-a876-4f08-ba68-4c5b821d2f8f None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:15.428123 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd4a5be-a876-4f08-ba68-4c5b821d2f8f None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:15.428123 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd4a5be-a876-4f08-ba68-4c5b821d2f8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:15.443034 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-7fd4a5be-a876-4f08-ba68-4c5b821d2f8f None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:15:15.444227 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7fd4a5be-a876-4f08-ba68-4c5b821d2f8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:15.510591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1437/2873] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:15:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:15:15.518492 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c185906-429d-4443-b7c6-da07edec1ec1 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:15.518872 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c185906-429d-4443-b7c6-da07edec1ec1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:15.519210 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c185906-429d-4443-b7c6-da07edec1ec1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:15.540831 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-4c185906-429d-4443-b7c6-da07edec1ec1 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:15:15.542093 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-4c185906-429d-4443-b7c6-da07edec1ec1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:15.551523 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1437/2874] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:15:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:15:15.555821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd4a5be-a876-4f08-ba68-4c5b821d2f8f None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:15.566567 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd4a5be-a876-4f08-ba68-4c5b821d2f8f None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:15.571443 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3784baa5-54df-4c19-b76a-2dc96e469973 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:15.572040 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3784baa5-54df-4c19-b76a-2dc96e469973 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:15.572320 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3784baa5-54df-4c19-b76a-2dc96e469973 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:15.572594 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3784baa5-54df-4c19-b76a-2dc96e469973 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:15.573307 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3784baa5-54df-4c19-b76a-2dc96e469973 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:15:15.573614 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3784baa5-54df-4c19-b76a-2dc96e469973 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:15.575236 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3784baa5-54df-4c19-b76a-2dc96e469973 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:15.587815 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1438/2875] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:15:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:15:15.591988 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c185906-429d-4443-b7c6-da07edec1ec1 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:15.602653 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c185906-429d-4443-b7c6-da07edec1ec1 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:15.607503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb656b9-5962-4118-b172-458811ac5fe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:15.608102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb656b9-5962-4118-b172-458811ac5fe0 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:15.608390 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb656b9-5962-4118-b172-458811ac5fe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:15.608663 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb656b9-5962-4118-b172-458811ac5fe0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:15.609357 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb656b9-5962-4118-b172-458811ac5fe0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:15.612190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb656b9-5962-4118-b172-458811ac5fe0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:15.712828 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1438/2876] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:15:15 2022] POST /identity/v3/projects => generated 383 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:15:15.717196 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3784baa5-54df-4c19-b76a-2dc96e469973 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:15.727837 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3784baa5-54df-4c19-b76a-2dc96e469973 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:15.786772 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1d2fe8-e37e-4290-bbbc-e85d5124493b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:15.787382 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1d2fe8-e37e-4290-bbbc-e85d5124493b admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:15.787688 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1d2fe8-e37e-4290-bbbc-e85d5124493b admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:15.787965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1d2fe8-e37e-4290-bbbc-e85d5124493b admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:15.788607 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1d2fe8-e37e-4290-bbbc-e85d5124493b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:15.790561 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f1d2fe8-e37e-4290-bbbc-e85d5124493b admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:15.911161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1439/2877] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:15:15 2022] POST /identity/v3/users => generated 292 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:15:15.915171 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb656b9-5962-4118-b172-458811ac5fe0 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:15.926800 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb656b9-5962-4118-b172-458811ac5fe0 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:15.933810 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56933e59-597a-4d72-8d69-4a2b27f1366b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:15.934639 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56933e59-597a-4d72-8d69-4a2b27f1366b admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:15.939815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56933e59-597a-4d72-8d69-4a2b27f1366b admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:15.939815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56933e59-597a-4d72-8d69-4a2b27f1366b admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:15.939815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56933e59-597a-4d72-8d69-4a2b27f1366b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:15.939815 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56933e59-597a-4d72-8d69-4a2b27f1366b admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:15.954717 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1439/2878] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:15:15 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:15:15.959721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1d2fe8-e37e-4290-bbbc-e85d5124493b admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:15.978394 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f1d2fe8-e37e-4290-bbbc-e85d5124493b admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:15.988818 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e383ef8-6ca7-4df0-9517-e26048daf156 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:15.989834 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e383ef8-6ca7-4df0-9517-e26048daf156 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:15.990319 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e383ef8-6ca7-4df0-9517-e26048daf156 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:15.991326 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e383ef8-6ca7-4df0-9517-e26048daf156 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:15.992097 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e383ef8-6ca7-4df0-9517-e26048daf156 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:15.995565 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e383ef8-6ca7-4df0-9517-e26048daf156 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:16.015842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1440/2879] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:15:15 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 04 01:15:16.019952 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56933e59-597a-4d72-8d69-4a2b27f1366b admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:16.030037 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56933e59-597a-4d72-8d69-4a2b27f1366b admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:16.035079 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5346442f-f1e1-4b3f-90db-ca45f869346e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:16.035743 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5346442f-f1e1-4b3f-90db-ca45f869346e admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:16.036025 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5346442f-f1e1-4b3f-90db-ca45f869346e admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:16.036302 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5346442f-f1e1-4b3f-90db-ca45f869346e admin admin] PATH_INFO: `/v3/projects/2215c39caa2847f8b7b69343f3a4278f/users/07355df444ce430093b1ae9e269f57fc/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:16.055209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5346442f-f1e1-4b3f-90db-ca45f869346e admin admin] RBAC: Authorizing `identity:create_grant(project_id=2215c39caa2847f8b7b69343f3a4278f, user_id=07355df444ce430093b1ae9e269f57fc, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:16.059702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5346442f-f1e1-4b3f-90db-ca45f869346e admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:16.122086 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1440/2880] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:15:16 2022] PUT /identity/v3/projects/2215c39caa2847f8b7b69343f3a4278f/users/07355df444ce430093b1ae9e269f57fc/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:15:16.134518 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376c9fef-28b9-45c4-b32c-370ef437f989 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:16.134814 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376c9fef-28b9-45c4-b32c-370ef437f989 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:16.135088 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376c9fef-28b9-45c4-b32c-370ef437f989 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:16.143658 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:15:16.163985 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-376c9fef-28b9-45c4-b32c-370ef437f989 None None] MFA Rules not processed for user `07355df444ce430093b1ae9e269f57fc`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:15:16.165315 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-376c9fef-28b9-45c4-b32c-370ef437f989 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:16.230611 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1441/2881] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:15:16 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:15:16.242881 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5346442f-f1e1-4b3f-90db-ca45f869346e admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:16.244855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-5346442f-f1e1-4b3f-90db-ca45f869346e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:16.305002 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5346442f-f1e1-4b3f-90db-ca45f869346e admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:16.308697 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad68ba48-fc62-45bd-871a-2c1ca2ede1e5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:16.309327 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad68ba48-fc62-45bd-871a-2c1ca2ede1e5 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:16.309609 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad68ba48-fc62-45bd-871a-2c1ca2ede1e5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:16.309883 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad68ba48-fc62-45bd-871a-2c1ca2ede1e5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:16.315930 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:15:16.319297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad68ba48-fc62-45bd-871a-2c1ca2ede1e5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:16.320753 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad68ba48-fc62-45bd-871a-2c1ca2ede1e5 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:16.330348 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1441/2882] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:15:16 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:15:16.349111 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376c9fef-28b9-45c4-b32c-370ef437f989 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:16.435793 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376c9fef-28b9-45c4-b32c-370ef437f989 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:16.449657 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b928a3a4-3017-4c82-9b3f-f94c7cc17d4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:16.451075 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b928a3a4-3017-4c82-9b3f-f94c7cc17d4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:16.451583 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b928a3a4-3017-4c82-9b3f-f94c7cc17d4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:16.451583 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b928a3a4-3017-4c82-9b3f-f94c7cc17d4e admin admin] PATH_INFO: `/v3/users/07355df444ce430093b1ae9e269f57fc` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:16.453021 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b928a3a4-3017-4c82-9b3f-f94c7cc17d4e admin admin] RBAC: Authorizing `identity:delete_user(user_id=07355df444ce430093b1ae9e269f57fc)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:16.456127 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b928a3a4-3017-4c82-9b3f-f94c7cc17d4e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:16.629205 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b928a3a4-3017-4c82-9b3f-f94c7cc17d4e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '07355df444ce430093b1ae9e269f57fc'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:16.706164 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b928a3a4-3017-4c82-9b3f-f94c7cc17d4e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '07355df444ce430093b1ae9e269f57fc'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:16.721668 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b928a3a4-3017-4c82-9b3f-f94c7cc17d4e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '07355df444ce430093b1ae9e269f57fc'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:16.722503 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b928a3a4-3017-4c82-9b3f-f94c7cc17d4e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '07355df444ce430093b1ae9e269f57fc'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:16.730324 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-b928a3a4-3017-4c82-9b3f-f94c7cc17d4e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '07355df444ce430093b1ae9e269f57fc'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:16.737329 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1442/2883] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:15:16 2022] DELETE /identity/v3/users/07355df444ce430093b1ae9e269f57fc => generated 0 bytes in 389 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:15:16.741979 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900ecadf-0677-43c4-953a-8a96e83a6d4b None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:16.742558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900ecadf-0677-43c4-953a-8a96e83a6d4b None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:16.742558 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900ecadf-0677-43c4-953a-8a96e83a6d4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:16.763306 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-900ecadf-0677-43c4-953a-8a96e83a6d4b None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:15:16.764512 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-900ecadf-0677-43c4-953a-8a96e83a6d4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:16.832172 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1442/2884] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:15:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:15:16.842908 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b928a3a4-3017-4c82-9b3f-f94c7cc17d4e admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:16.845388 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-b928a3a4-3017-4c82-9b3f-f94c7cc17d4e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:16.908863 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:15:16.921774 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b928a3a4-3017-4c82-9b3f-f94c7cc17d4e admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:16.926645 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a933707e-f99b-4fc2-b19d-e73d34741b4c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:16.927434 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a933707e-f99b-4fc2-b19d-e73d34741b4c service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:16.927705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a933707e-f99b-4fc2-b19d-e73d34741b4c service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:16.928291 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a933707e-f99b-4fc2-b19d-e73d34741b4c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:16.940123 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a933707e-f99b-4fc2-b19d-e73d34741b4c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:16.943860 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a933707e-f99b-4fc2-b19d-e73d34741b4c service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:16.951702 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1443/2885] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:15:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:15:17.449555 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900ecadf-0677-43c4-953a-8a96e83a6d4b None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:17.451019 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-900ecadf-0677-43c4-953a-8a96e83a6d4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:17.464383 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900ecadf-0677-43c4-953a-8a96e83a6d4b None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:17.470012 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1080012f-dd23-4841-9dd4-99f970e298a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:17.470698 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1080012f-dd23-4841-9dd4-99f970e298a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:17.470978 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1080012f-dd23-4841-9dd4-99f970e298a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:17.471252 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1080012f-dd23-4841-9dd4-99f970e298a0 admin admin] PATH_INFO: `/v3/projects/2215c39caa2847f8b7b69343f3a4278f` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:17.472449 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1080012f-dd23-4841-9dd4-99f970e298a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2215c39caa2847f8b7b69343f3a4278f)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:17.474413 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1080012f-dd23-4841-9dd4-99f970e298a0 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:17.598277 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-1080012f-dd23-4841-9dd4-99f970e298a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2215c39caa2847f8b7b69343f3a4278f', 'request_id': 'req-1080012f-dd23-4841-9dd4-99f970e298a0'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:17.603781 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-1080012f-dd23-4841-9dd4-99f970e298a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2215c39caa2847f8b7b69343f3a4278f', 'request_id': 'req-1080012f-dd23-4841-9dd4-99f970e298a0'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:17.688117 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-1080012f-dd23-4841-9dd4-99f970e298a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2215c39caa2847f8b7b69343f3a4278f', 'request_id': 'req-1080012f-dd23-4841-9dd4-99f970e298a0'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:17.703638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-1080012f-dd23-4841-9dd4-99f970e298a0 admin admin] The token cache is being invalidate because project 2215c39caa2847f8b7b69343f3a4278f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:15:17.703933 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-1080012f-dd23-4841-9dd4-99f970e298a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:17.704645 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-1080012f-dd23-4841-9dd4-99f970e298a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:17.706206 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1443/2886] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:15:17 2022] DELETE /identity/v3/projects/2215c39caa2847f8b7b69343f3a4278f => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:15:17.711192 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f20aea-12aa-44cc-a642-12be5522ca8d None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:17.711705 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f20aea-12aa-44cc-a642-12be5522ca8d None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:17.711984 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f20aea-12aa-44cc-a642-12be5522ca8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:17.733177 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-d1f20aea-12aa-44cc-a642-12be5522ca8d None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:15:17.734772 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-d1f20aea-12aa-44cc-a642-12be5522ca8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:17.806150 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1444/2887] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:15:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:15:17.814532 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86a1a09-796c-46be-b68f-9fd5694a3f89 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:17.814729 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86a1a09-796c-46be-b68f-9fd5694a3f89 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:17.815001 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86a1a09-796c-46be-b68f-9fd5694a3f89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:17.845625 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-e86a1a09-796c-46be-b68f-9fd5694a3f89 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:15:17.847003 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-e86a1a09-796c-46be-b68f-9fd5694a3f89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:17.860450 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1444/2888] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:15:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:15:17.861748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f20aea-12aa-44cc-a642-12be5522ca8d None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:17.872172 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f20aea-12aa-44cc-a642-12be5522ca8d None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:17.877574 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c504b83f-2ba3-4294-877e-a6a4170e3189 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:17.878162 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c504b83f-2ba3-4294-877e-a6a4170e3189 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:17.878479 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c504b83f-2ba3-4294-877e-a6a4170e3189 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:17.878753 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c504b83f-2ba3-4294-877e-a6a4170e3189 admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:17.879536 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c504b83f-2ba3-4294-877e-a6a4170e3189 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:15:17.879727 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c504b83f-2ba3-4294-877e-a6a4170e3189 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:17.881451 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c504b83f-2ba3-4294-877e-a6a4170e3189 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:17.893605 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1445/2889] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:15:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:15:17.898420 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86a1a09-796c-46be-b68f-9fd5694a3f89 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:17.908956 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86a1a09-796c-46be-b68f-9fd5694a3f89 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:17.914757 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f617b3f-b220-4251-8ae5-f4f1f8ff2777 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:17.915360 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f617b3f-b220-4251-8ae5-f4f1f8ff2777 admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:17.915641 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f617b3f-b220-4251-8ae5-f4f1f8ff2777 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:17.915945 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f617b3f-b220-4251-8ae5-f4f1f8ff2777 admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:17.916586 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f617b3f-b220-4251-8ae5-f4f1f8ff2777 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:17.918527 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f617b3f-b220-4251-8ae5-f4f1f8ff2777 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:18.009013 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1445/2890] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:15:17 2022] POST /identity/v3/projects => generated 361 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:15:18.010463 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c504b83f-2ba3-4294-877e-a6a4170e3189 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:18.021609 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c504b83f-2ba3-4294-877e-a6a4170e3189 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:18.083935 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92f9955-1ae6-4710-9349-34d4ccef9d72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:18.084556 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92f9955-1ae6-4710-9349-34d4ccef9d72 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:18.084834 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92f9955-1ae6-4710-9349-34d4ccef9d72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:18.085197 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92f9955-1ae6-4710-9349-34d4ccef9d72 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:18.085847 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92f9955-1ae6-4710-9349-34d4ccef9d72 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:18.087813 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92f9955-1ae6-4710-9349-34d4ccef9d72 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:18.197790 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1446/2891] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:15:18 2022] POST /identity/v3/users => generated 281 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 04 01:15:18.202258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f617b3f-b220-4251-8ae5-f4f1f8ff2777 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:18.213905 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f617b3f-b220-4251-8ae5-f4f1f8ff2777 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:18.220398 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc1bd32-5a03-4d05-bb12-f32ae2127732 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:18.221087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1bd32-5a03-4d05-bb12-f32ae2127732 admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:18.221392 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1bd32-5a03-4d05-bb12-f32ae2127732 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:18.221668 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc1bd32-5a03-4d05-bb12-f32ae2127732 admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:18.222337 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc1bd32-5a03-4d05-bb12-f32ae2127732 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:18.223964 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc1bd32-5a03-4d05-bb12-f32ae2127732 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:18.238910 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1446/2892] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:15:18 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:15:18.240182 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92f9955-1ae6-4710-9349-34d4ccef9d72 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:18.251533 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92f9955-1ae6-4710-9349-34d4ccef9d72 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:18.256591 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a544646e-a980-4050-a651-5d3a4f70b079 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:18.257733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a544646e-a980-4050-a651-5d3a4f70b079 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:18.257733 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a544646e-a980-4050-a651-5d3a4f70b079 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:18.257977 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a544646e-a980-4050-a651-5d3a4f70b079 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:18.258418 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a544646e-a980-4050-a651-5d3a4f70b079 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:18.260046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a544646e-a980-4050-a651-5d3a4f70b079 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:18.275838 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1447/2893] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:15:18 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:15:18.277370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc1bd32-5a03-4d05-bb12-f32ae2127732 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:18.290768 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc1bd32-5a03-4d05-bb12-f32ae2127732 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:18.295859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420022b5-9407-41d6-887d-3d5cf50c39c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:18.297265 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420022b5-9407-41d6-887d-3d5cf50c39c5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:18.297865 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420022b5-9407-41d6-887d-3d5cf50c39c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:18.298355 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420022b5-9407-41d6-887d-3d5cf50c39c5 admin admin] PATH_INFO: `/v3/projects/366b418257664a79b3c1d3f373c7ac5b/users/2d5958a53dda46ca887c874e01682a5e/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:18.324638 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420022b5-9407-41d6-887d-3d5cf50c39c5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=366b418257664a79b3c1d3f373c7ac5b, user_id=2d5958a53dda46ca887c874e01682a5e, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:18.330303 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420022b5-9407-41d6-887d-3d5cf50c39c5 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:18.411343 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1447/2894] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:15:18 2022] PUT /identity/v3/projects/366b418257664a79b3c1d3f373c7ac5b/users/2d5958a53dda46ca887c874e01682a5e/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:15:18.422270 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7730690a-deab-4885-b1d8-3b4167949702 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:18.422474 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7730690a-deab-4885-b1d8-3b4167949702 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:18.422840 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7730690a-deab-4885-b1d8-3b4167949702 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:18.462429 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-7730690a-deab-4885-b1d8-3b4167949702 None None] MFA Rules not processed for user `2d5958a53dda46ca887c874e01682a5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:15:18.464188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7730690a-deab-4885-b1d8-3b4167949702 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:18.478170 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:15:18.544891 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1448/2895] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:15:18 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:15:18.581994 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420022b5-9407-41d6-887d-3d5cf50c39c5 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:18.583842 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-420022b5-9407-41d6-887d-3d5cf50c39c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:18.598887 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:15:18.643113 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420022b5-9407-41d6-887d-3d5cf50c39c5 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:18.646979 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed30901e-a6ca-4457-a431-aad4d69430f7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:18.647595 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed30901e-a6ca-4457-a431-aad4d69430f7 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:18.647878 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed30901e-a6ca-4457-a431-aad4d69430f7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:18.648155 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed30901e-a6ca-4457-a431-aad4d69430f7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:18.658820 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed30901e-a6ca-4457-a431-aad4d69430f7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:18.659788 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed30901e-a6ca-4457-a431-aad4d69430f7 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:18.670971 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1448/2896] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:15:18 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:15:18.687791 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7730690a-deab-4885-b1d8-3b4167949702 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:18.750788 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7730690a-deab-4885-b1d8-3b4167949702 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:18.755349 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441c8bae-9f36-4ef1-84be-2b2a7bfd1e36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:18.756002 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441c8bae-9f36-4ef1-84be-2b2a7bfd1e36 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:18.756278 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441c8bae-9f36-4ef1-84be-2b2a7bfd1e36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:18.756552 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441c8bae-9f36-4ef1-84be-2b2a7bfd1e36 admin admin] PATH_INFO: `/v3/users/2d5958a53dda46ca887c874e01682a5e` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:18.757694 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441c8bae-9f36-4ef1-84be-2b2a7bfd1e36 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d5958a53dda46ca887c874e01682a5e)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:18.759612 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441c8bae-9f36-4ef1-84be-2b2a7bfd1e36 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:18.987493 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-441c8bae-9f36-4ef1-84be-2b2a7bfd1e36 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d5958a53dda46ca887c874e01682a5e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:19.048752 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-441c8bae-9f36-4ef1-84be-2b2a7bfd1e36 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d5958a53dda46ca887c874e01682a5e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:19.062197 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-441c8bae-9f36-4ef1-84be-2b2a7bfd1e36 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d5958a53dda46ca887c874e01682a5e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:19.064160 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-441c8bae-9f36-4ef1-84be-2b2a7bfd1e36 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d5958a53dda46ca887c874e01682a5e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:19.073115 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-441c8bae-9f36-4ef1-84be-2b2a7bfd1e36 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d5958a53dda46ca887c874e01682a5e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:19.079503 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1449/2897] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:15:18 2022] DELETE /identity/v3/users/2d5958a53dda46ca887c874e01682a5e => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:15:19.084004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab73475-c399-46a2-a08b-3dd2afcb2562 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:19.084291 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab73475-c399-46a2-a08b-3dd2afcb2562 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:19.084566 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab73475-c399-46a2-a08b-3dd2afcb2562 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:19.105693 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-aab73475-c399-46a2-a08b-3dd2afcb2562 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:15:19.106826 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-aab73475-c399-46a2-a08b-3dd2afcb2562 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:19.168219 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1449/2898] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:15:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:15:19.178367 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441c8bae-9f36-4ef1-84be-2b2a7bfd1e36 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:19.179804 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-441c8bae-9f36-4ef1-84be-2b2a7bfd1e36 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:19.214840 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:15:19.237667 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441c8bae-9f36-4ef1-84be-2b2a7bfd1e36 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:19.241463 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd1a802-f5a9-40ee-a558-b36e84e8ea87 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:19.242065 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd1a802-f5a9-40ee-a558-b36e84e8ea87 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:19.242344 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd1a802-f5a9-40ee-a558-b36e84e8ea87 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:19.242619 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd1a802-f5a9-40ee-a558-b36e84e8ea87 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:19.251841 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd1a802-f5a9-40ee-a558-b36e84e8ea87 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:19.253360 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd1a802-f5a9-40ee-a558-b36e84e8ea87 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:19.263247 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1450/2899] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:15:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:15:19.685782 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab73475-c399-46a2-a08b-3dd2afcb2562 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:19.687521 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-aab73475-c399-46a2-a08b-3dd2afcb2562 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:19.702220 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab73475-c399-46a2-a08b-3dd2afcb2562 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:19.707258 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc43c3c-91b9-4549-bafa-d2c1fed9c47f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:19.708033 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc43c3c-91b9-4549-bafa-d2c1fed9c47f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:19.708312 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc43c3c-91b9-4549-bafa-d2c1fed9c47f admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:19.708583 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc43c3c-91b9-4549-bafa-d2c1fed9c47f admin admin] PATH_INFO: `/v3/projects/366b418257664a79b3c1d3f373c7ac5b` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:19.709854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc43c3c-91b9-4549-bafa-d2c1fed9c47f admin admin] RBAC: Authorizing `identity:delete_project(project_id=366b418257664a79b3c1d3f373c7ac5b)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:19.711785 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc43c3c-91b9-4549-bafa-d2c1fed9c47f admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:19.828088 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0fc43c3c-91b9-4549-bafa-d2c1fed9c47f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '366b418257664a79b3c1d3f373c7ac5b', 'request_id': 'req-0fc43c3c-91b9-4549-bafa-d2c1fed9c47f'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:19.835192 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0fc43c3c-91b9-4549-bafa-d2c1fed9c47f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '366b418257664a79b3c1d3f373c7ac5b', 'request_id': 'req-0fc43c3c-91b9-4549-bafa-d2c1fed9c47f'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:19.927504 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0fc43c3c-91b9-4549-bafa-d2c1fed9c47f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '366b418257664a79b3c1d3f373c7ac5b', 'request_id': 'req-0fc43c3c-91b9-4549-bafa-d2c1fed9c47f'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:19.941913 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0fc43c3c-91b9-4549-bafa-d2c1fed9c47f admin admin] The token cache is being invalidate because project 366b418257664a79b3c1d3f373c7ac5b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:15:19.942209 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0fc43c3c-91b9-4549-bafa-d2c1fed9c47f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:19.942877 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0fc43c3c-91b9-4549-bafa-d2c1fed9c47f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:19.944885 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1450/2900] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:15:19 2022] DELETE /identity/v3/projects/366b418257664a79b3c1d3f373c7ac5b => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:15:19.950470 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497f75bf-e669-40dc-8d54-47ebccc246ae None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:19.950965 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497f75bf-e669-40dc-8d54-47ebccc246ae None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:19.951101 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497f75bf-e669-40dc-8d54-47ebccc246ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:19.972219 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-497f75bf-e669-40dc-8d54-47ebccc246ae None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:15:19.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-497f75bf-e669-40dc-8d54-47ebccc246ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:20.039351 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1451/2901] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:15:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:15:20.049808 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74bf57e-16fb-4552-8587-2a5eb7f66d61 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:20.050087 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74bf57e-16fb-4552-8587-2a5eb7f66d61 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:20.050354 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74bf57e-16fb-4552-8587-2a5eb7f66d61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:20.073868 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-d74bf57e-16fb-4552-8587-2a5eb7f66d61 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:15:20.075125 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d74bf57e-16fb-4552-8587-2a5eb7f66d61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:20.083687 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1451/2902] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:15:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:15:20.087930 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497f75bf-e669-40dc-8d54-47ebccc246ae None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:20.100444 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497f75bf-e669-40dc-8d54-47ebccc246ae None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:20.107299 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b12378-dbf2-4377-983a-e3dadd28f8fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:20.107913 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b12378-dbf2-4377-983a-e3dadd28f8fa admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:20.108188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b12378-dbf2-4377-983a-e3dadd28f8fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:20.108459 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b12378-dbf2-4377-983a-e3dadd28f8fa admin admin] PATH_INFO: `/v3/domains` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:20.109139 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b12378-dbf2-4377-983a-e3dadd28f8fa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81558) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 04 01:15:20.109447 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b12378-dbf2-4377-983a-e3dadd28f8fa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:20.111112 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b12378-dbf2-4377-983a-e3dadd28f8fa admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:20.123586 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1452/2903] 10.222.0.42 () {58 vars in 1153 bytes} [Sun Sep 4 01:15:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 04 01:15:20.128615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74bf57e-16fb-4552-8587-2a5eb7f66d61 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:20.138832 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74bf57e-16fb-4552-8587-2a5eb7f66d61 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:20.144834 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4437735-dc3b-4a8b-8f4c-a83d897031ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:20.145481 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4437735-dc3b-4a8b-8f4c-a83d897031ea admin admin] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:20.145895 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4437735-dc3b-4a8b-8f4c-a83d897031ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:20.146396 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4437735-dc3b-4a8b-8f4c-a83d897031ea admin admin] PATH_INFO: `/v3/projects` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:20.147404 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4437735-dc3b-4a8b-8f4c-a83d897031ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:20.149992 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4437735-dc3b-4a8b-8f4c-a83d897031ea admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:20.249761 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1452/2904] 10.222.0.42 () {60 vars in 1153 bytes} [Sun Sep 4 01:15:20 2022] POST /identity/v3/projects => generated 371 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:15:20.255230 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b12378-dbf2-4377-983a-e3dadd28f8fa admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:20.265647 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b12378-dbf2-4377-983a-e3dadd28f8fa admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:20.339467 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2178e25-1723-428c-9616-5fe3a7b95c28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:20.340137 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2178e25-1723-428c-9616-5fe3a7b95c28 admin admin] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:20.340353 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2178e25-1723-428c-9616-5fe3a7b95c28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:20.340623 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2178e25-1723-428c-9616-5fe3a7b95c28 admin admin] PATH_INFO: `/v3/users` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:20.341304 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2178e25-1723-428c-9616-5fe3a7b95c28 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:20.343327 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2178e25-1723-428c-9616-5fe3a7b95c28 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:20.466081 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1453/2905] 10.222.0.42 () {60 vars in 1144 bytes} [Sun Sep 4 01:15:20 2022] POST /identity/v3/users => generated 286 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 04 01:15:20.470214 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4437735-dc3b-4a8b-8f4c-a83d897031ea admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:20.480370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4437735-dc3b-4a8b-8f4c-a83d897031ea admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:20.484918 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be78a45-74bf-4bc8-ac1e-6ceac664cbdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:20.486060 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be78a45-74bf-4bc8-ac1e-6ceac664cbdb admin admin] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:20.486331 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be78a45-74bf-4bc8-ac1e-6ceac664cbdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:20.486647 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be78a45-74bf-4bc8-ac1e-6ceac664cbdb admin admin] PATH_INFO: `/v3/roles` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:20.487290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be78a45-74bf-4bc8-ac1e-6ceac664cbdb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:20.488880 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be78a45-74bf-4bc8-ac1e-6ceac664cbdb admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:20.494761 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:15:20.500103 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1453/2906] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:15:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:15:20.504821 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2178e25-1723-428c-9616-5fe3a7b95c28 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:20.515287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2178e25-1723-428c-9616-5fe3a7b95c28 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:20.521363 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6884e6-c504-4193-a6dd-cf435f13aa89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:20.521964 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6884e6-c504-4193-a6dd-cf435f13aa89 admin admin] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:20.522242 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6884e6-c504-4193-a6dd-cf435f13aa89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:20.522512 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6884e6-c504-4193-a6dd-cf435f13aa89 admin admin] PATH_INFO: `/v3/roles` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:20.523167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6884e6-c504-4193-a6dd-cf435f13aa89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:20.524759 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6884e6-c504-4193-a6dd-cf435f13aa89 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:20.538802 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1454/2907] 10.222.0.42 () {58 vars in 1122 bytes} [Sun Sep 4 01:15:20 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 04 01:15:20.541737 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be78a45-74bf-4bc8-ac1e-6ceac664cbdb admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:20.551924 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be78a45-74bf-4bc8-ac1e-6ceac664cbdb admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:20.556528 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70a476c-1a9e-4679-8a1d-975d072f4e0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:20.557218 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70a476c-1a9e-4679-8a1d-975d072f4e0d admin admin] REQUEST_METHOD: `PUT` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:20.557503 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70a476c-1a9e-4679-8a1d-975d072f4e0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:20.557780 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70a476c-1a9e-4679-8a1d-975d072f4e0d admin admin] PATH_INFO: `/v3/projects/83bd5ff9f21f4913b2848324640531e2/users/fb9d384318f6461a8efea43aebea2ed9/roles/4bfd6d0544ee4a9e933538e78135a12c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:20.572764 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70a476c-1a9e-4679-8a1d-975d072f4e0d admin admin] RBAC: Authorizing `identity:create_grant(project_id=83bd5ff9f21f4913b2848324640531e2, user_id=fb9d384318f6461a8efea43aebea2ed9, role_id=4bfd6d0544ee4a9e933538e78135a12c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:20.575448 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70a476c-1a9e-4679-8a1d-975d072f4e0d admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:20.653628 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1454/2908] 10.222.0.42 () {60 vars in 1483 bytes} [Sun Sep 4 01:15:20 2022] PUT /identity/v3/projects/83bd5ff9f21f4913b2848324640531e2/users/fb9d384318f6461a8efea43aebea2ed9/roles/4bfd6d0544ee4a9e933538e78135a12c => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:15:20.667808 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae42b3be-c448-4413-b880-35de00aaef32 None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:20.668093 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae42b3be-c448-4413-b880-35de00aaef32 None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:20.668367 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae42b3be-c448-4413-b880-35de00aaef32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:20.697046 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-ae42b3be-c448-4413-b880-35de00aaef32 None None] MFA Rules not processed for user `fb9d384318f6461a8efea43aebea2ed9`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:15:20.698447 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ae42b3be-c448-4413-b880-35de00aaef32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:20.730763 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:15:20.771642 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1455/2909] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:15:20 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:15:20.783979 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70a476c-1a9e-4679-8a1d-975d072f4e0d admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:20.785790 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-d70a476c-1a9e-4679-8a1d-975d072f4e0d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:20.844447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70a476c-1a9e-4679-8a1d-975d072f4e0d admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:20.849130 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c86f1b-bf99-4639-a449-20f8f0b221e5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '615a607b6be14b82979f94eabaf4c34c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:20.849751 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c86f1b-bf99-4639-a449-20f8f0b221e5 service cinder] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:20.850032 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c86f1b-bf99-4639-a449-20f8f0b221e5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:20.850303 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c86f1b-bf99-4639-a449-20f8f0b221e5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:20.863450 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c86f1b-bf99-4639-a449-20f8f0b221e5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:20.865036 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c86f1b-bf99-4639-a449-20f8f0b221e5 service cinder] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:20.875723 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1455/2910] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:15:20 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:15:20.901829 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae42b3be-c448-4413-b880-35de00aaef32 None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:20.973925 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae42b3be-c448-4413-b880-35de00aaef32 None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:20.978820 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcd0ee4-15f8-495a-a5b1-c232faf83921 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:20.979547 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcd0ee4-15f8-495a-a5b1-c232faf83921 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:20.979822 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcd0ee4-15f8-495a-a5b1-c232faf83921 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:20.980094 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcd0ee4-15f8-495a-a5b1-c232faf83921 admin admin] PATH_INFO: `/v3/users/fb9d384318f6461a8efea43aebea2ed9` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:20.981295 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcd0ee4-15f8-495a-a5b1-c232faf83921 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb9d384318f6461a8efea43aebea2ed9)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:20.983225 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcd0ee4-15f8-495a-a5b1-c232faf83921 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:21.169245 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bfcd0ee4-15f8-495a-a5b1-c232faf83921 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb9d384318f6461a8efea43aebea2ed9'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:21.275284 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bfcd0ee4-15f8-495a-a5b1-c232faf83921 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb9d384318f6461a8efea43aebea2ed9'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:21.287770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bfcd0ee4-15f8-495a-a5b1-c232faf83921 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb9d384318f6461a8efea43aebea2ed9'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:21.288413 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bfcd0ee4-15f8-495a-a5b1-c232faf83921 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb9d384318f6461a8efea43aebea2ed9'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:21.295348 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-bfcd0ee4-15f8-495a-a5b1-c232faf83921 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb9d384318f6461a8efea43aebea2ed9'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:21.301836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1456/2911] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:15:20 2022] DELETE /identity/v3/users/fb9d384318f6461a8efea43aebea2ed9 => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:15:21.306632 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811a757b-83a1-4758-92d7-68ecc5d018e1 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:21.306917 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811a757b-83a1-4758-92d7-68ecc5d018e1 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:21.307192 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811a757b-83a1-4758-92d7-68ecc5d018e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:21.326766 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-811a757b-83a1-4758-92d7-68ecc5d018e1 None None] MFA Rules not processed for user `01782590dd294bd488a6d5c08a043ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:15:21.327958 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-811a757b-83a1-4758-92d7-68ecc5d018e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:21.385826 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1456/2912] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:15:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 04 01:15:21.395707 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcd0ee4-15f8-495a-a5b1-c232faf83921 admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:21.397122 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-bfcd0ee4-15f8-495a-a5b1-c232faf83921 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:21.450188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:15:21.454034 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcd0ee4-15f8-495a-a5b1-c232faf83921 admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:21.457701 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0838e8-51fa-401c-b60c-20cb5b805e79 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:21.458311 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0838e8-51fa-401c-b60c-20cb5b805e79 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:21.460857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0838e8-51fa-401c-b60c-20cb5b805e79 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:21.460857 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0838e8-51fa-401c-b60c-20cb5b805e79 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:21.467982 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0838e8-51fa-401c-b60c-20cb5b805e79 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:21.469534 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0838e8-51fa-401c-b60c-20cb5b805e79 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:21.478883 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1457/2913] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:15:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:15:21.911447 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811a757b-83a1-4758-92d7-68ecc5d018e1 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:21.913393 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-811a757b-83a1-4758-92d7-68ecc5d018e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:21.926960 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811a757b-83a1-4758-92d7-68ecc5d018e1 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:21.932775 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182a305c-7280-42e9-8409-4adcc386c566 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:21.933574 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182a305c-7280-42e9-8409-4adcc386c566 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:21.933855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182a305c-7280-42e9-8409-4adcc386c566 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:21.934119 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182a305c-7280-42e9-8409-4adcc386c566 admin admin] PATH_INFO: `/v3/projects/83bd5ff9f21f4913b2848324640531e2` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:21.935283 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182a305c-7280-42e9-8409-4adcc386c566 admin admin] RBAC: Authorizing `identity:delete_project(project_id=83bd5ff9f21f4913b2848324640531e2)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:21.937207 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182a305c-7280-42e9-8409-4adcc386c566 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:22.057340 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-182a305c-7280-42e9-8409-4adcc386c566 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '83bd5ff9f21f4913b2848324640531e2', 'request_id': 'req-182a305c-7280-42e9-8409-4adcc386c566'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:22.064216 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-182a305c-7280-42e9-8409-4adcc386c566 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '83bd5ff9f21f4913b2848324640531e2', 'request_id': 'req-182a305c-7280-42e9-8409-4adcc386c566'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:22.162628 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-182a305c-7280-42e9-8409-4adcc386c566 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '83bd5ff9f21f4913b2848324640531e2', 'request_id': 'req-182a305c-7280-42e9-8409-4adcc386c566'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:22.176074 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-182a305c-7280-42e9-8409-4adcc386c566 admin admin] The token cache is being invalidate because project 83bd5ff9f21f4913b2848324640531e2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:15:22.176365 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-182a305c-7280-42e9-8409-4adcc386c566 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:22.176987 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-182a305c-7280-42e9-8409-4adcc386c566 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:15:22.178606 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1457/2914] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:15:21 2022] DELETE /identity/v3/projects/83bd5ff9f21f4913b2848324640531e2 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 04 01:15:23.459214 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0838e8-51fa-401c-b60c-20cb5b805e79 service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:23.468167 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0838e8-51fa-401c-b60c-20cb5b805e79 service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:23.517687 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e8c13b-3fd4-4e3e-b9c3-88399ed3960c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:23.518311 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e8c13b-3fd4-4e3e-b9c3-88399ed3960c service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:23.518592 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e8c13b-3fd4-4e3e-b9c3-88399ed3960c service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:23.518866 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e8c13b-3fd4-4e3e-b9c3-88399ed3960c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:23.520624 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-26e8c13b-3fd4-4e3e-b9c3-88399ed3960c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:23.583178 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e8c13b-3fd4-4e3e-b9c3-88399ed3960c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:23.584618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e8c13b-3fd4-4e3e-b9c3-88399ed3960c service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:23.594071 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1458/2915] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:15:23 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:15:32.171508 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:15:33.589846 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=54 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:15:39.493219 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182a305c-7280-42e9-8409-4adcc386c566 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:15:39.494738 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-182a305c-7280-42e9-8409-4adcc386c566 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:39.507652 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182a305c-7280-42e9-8409-4adcc386c566 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:15:39.511255 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35203913-4812-488c-b93d-31848cda7833 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:15:39.511854 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35203913-4812-488c-b93d-31848cda7833 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:15:39.512136 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35203913-4812-488c-b93d-31848cda7833 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:15:39.512411 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35203913-4812-488c-b93d-31848cda7833 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:15:39.514333 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-35203913-4812-488c-b93d-31848cda7833 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:15:39.524464 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35203913-4812-488c-b93d-31848cda7833 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:15:39.525922 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35203913-4812-488c-b93d-31848cda7833 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:15:39.533954 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1458/2916] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:15:39 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:15:49.530339 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=6 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:16:30.465108 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e8c13b-3fd4-4e3e-b9c3-88399ed3960c service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:16:30.467618 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-26e8c13b-3fd4-4e3e-b9c3-88399ed3960c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:16:30.576836 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e8c13b-3fd4-4e3e-b9c3-88399ed3960c service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:16:30.582805 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28417a86-1468-428e-ba70-dcbb79deb834 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:16:30.583633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28417a86-1468-428e-ba70-dcbb79deb834 service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:16:30.583917 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28417a86-1468-428e-ba70-dcbb79deb834 service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:16:30.584192 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28417a86-1468-428e-ba70-dcbb79deb834 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:16:30.586143 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-28417a86-1468-428e-ba70-dcbb79deb834 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:16:30.600263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28417a86-1468-428e-ba70-dcbb79deb834 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:16:30.601842 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28417a86-1468-428e-ba70-dcbb79deb834 service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:16:30.624868 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1459/2917] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:16:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:16:40.000184 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8678e2-8ebc-4f9e-8d60-346983a30096 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:16:40.000680 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8678e2-8ebc-4f9e-8d60-346983a30096 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:16:40.000797 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8678e2-8ebc-4f9e-8d60-346983a30096 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:16:40.050540 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-da8678e2-8ebc-4f9e-8d60-346983a30096 None None] MFA Rules not processed for user `70cef7c271e046a3b49230057cc2d28f`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:16:40.051755 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-da8678e2-8ebc-4f9e-8d60-346983a30096 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:16:40.114959 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1459/2918] 10.222.0.42 () {58 vars in 967 bytes} [Sun Sep 4 01:16:39 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:16:40.155427 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28417a86-1468-428e-ba70-dcbb79deb834 service nova] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:16:40.156835 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-28417a86-1468-428e-ba70-dcbb79deb834 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:16:40.246652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28417a86-1468-428e-ba70-dcbb79deb834 service nova] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:16:40.253967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eb32fb-75aa-4327-95f1-44b84b31a0b7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '123ab6ff796c4e599ce1d6f4095bd898', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:16:40.256902 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb32fb-75aa-4327-95f1-44b84b31a0b7 service swift] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:16:40.257229 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb32fb-75aa-4327-95f1-44b84b31a0b7 service swift] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:16:40.257508 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb32fb-75aa-4327-95f1-44b84b31a0b7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:16:40.268721 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eb32fb-75aa-4327-95f1-44b84b31a0b7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:16:40.270332 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eb32fb-75aa-4327-95f1-44b84b31a0b7 service swift] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:16:40.280904 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1460/2919] 10.222.0.42 () {60 vars in 1384 bytes} [Sun Sep 4 01:16:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 04 01:16:44.285550 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8678e2-8ebc-4f9e-8d60-346983a30096 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:16:44.287010 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-da8678e2-8ebc-4f9e-8d60-346983a30096 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:16:44.356936 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8678e2-8ebc-4f9e-8d60-346983a30096 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:16:44.362028 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c939eb02-d971-4cab-9f42-203eb5004296 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b29f444d56ec49308f7fef16f4d1eea5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:16:44.362615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c939eb02-d971-4cab-9f42-203eb5004296 service placement] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:16:44.362899 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c939eb02-d971-4cab-9f42-203eb5004296 service placement] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:16:44.363171 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c939eb02-d971-4cab-9f42-203eb5004296 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:16:44.364894 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-c939eb02-d971-4cab-9f42-203eb5004296 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:16:44.377456 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c939eb02-d971-4cab-9f42-203eb5004296 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:16:44.379004 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c939eb02-d971-4cab-9f42-203eb5004296 service placement] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:16:44.390608 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1460/2920] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:16:44 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:16:48.042621 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eb32fb-75aa-4327-95f1-44b84b31a0b7 service swift] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:16:48.044223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-89eb32fb-75aa-4327-95f1-44b84b31a0b7 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:16:48.104383 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=100 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:16:48.110181 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eb32fb-75aa-4327-95f1-44b84b31a0b7 service swift] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:16:48.114653 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4864bf5c-6580-4749-bb27-9d8e79556c6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:16:48.115273 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4864bf5c-6580-4749-bb27-9d8e79556c6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:16:48.115551 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4864bf5c-6580-4749-bb27-9d8e79556c6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:16:48.115823 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4864bf5c-6580-4749-bb27-9d8e79556c6e admin admin] PATH_INFO: `/v3/users/9b5efa37b94b47949344d00ff88be4fb` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:16:48.116939 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4864bf5c-6580-4749-bb27-9d8e79556c6e admin admin] RBAC: Authorizing `identity:delete_user(user_id=9b5efa37b94b47949344d00ff88be4fb)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:16:48.119593 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4864bf5c-6580-4749-bb27-9d8e79556c6e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:16:48.287948 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4864bf5c-6580-4749-bb27-9d8e79556c6e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9b5efa37b94b47949344d00ff88be4fb'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:48.348442 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4864bf5c-6580-4749-bb27-9d8e79556c6e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9b5efa37b94b47949344d00ff88be4fb'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:48.373748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4864bf5c-6580-4749-bb27-9d8e79556c6e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9b5efa37b94b47949344d00ff88be4fb'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:48.373748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4864bf5c-6580-4749-bb27-9d8e79556c6e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9b5efa37b94b47949344d00ff88be4fb'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:48.373748 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-4864bf5c-6580-4749-bb27-9d8e79556c6e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9b5efa37b94b47949344d00ff88be4fb'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:48.374589 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1461/2921] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:16:48 2022] DELETE /identity/v3/users/9b5efa37b94b47949344d00ff88be4fb => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:16:48.643949 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c939eb02-d971-4cab-9f42-203eb5004296 service placement] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:16:48.655674 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c939eb02-d971-4cab-9f42-203eb5004296 service placement] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:16:48.718214 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:16:48.718855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:16:48.719137 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:16:48.719414 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237 admin admin] PATH_INFO: `/v3/projects/027792a6118140099aaf522ca15bb07e` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:16:48.720541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237 admin admin] RBAC: Authorizing `identity:delete_project(project_id=027792a6118140099aaf522ca15bb07e)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:16:48.722517 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:16:48.724926 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=100 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:16:48.831065 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '027792a6118140099aaf522ca15bb07e', 'request_id': 'req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:48.836536 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '027792a6118140099aaf522ca15bb07e', 'request_id': 'req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:48.908847 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '027792a6118140099aaf522ca15bb07e', 'request_id': 'req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:48.920648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237 admin admin] The token cache is being invalidate because project 027792a6118140099aaf522ca15bb07e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:16:48.920945 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:48.921621 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:48.923083 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1461/2922] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:16:48 2022] DELETE /identity/v3/projects/027792a6118140099aaf522ca15bb07e => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:16:48.927276 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4864bf5c-6580-4749-bb27-9d8e79556c6e admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:16:48.928709 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4864bf5c-6580-4749-bb27-9d8e79556c6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:16:48.942494 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4864bf5c-6580-4749-bb27-9d8e79556c6e admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:16:48.999864 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958a8dc1-0f49-4b2f-afdd-23b4cbab7a30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:16:49.000503 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958a8dc1-0f49-4b2f-afdd-23b4cbab7a30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:16:49.000789 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958a8dc1-0f49-4b2f-afdd-23b4cbab7a30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:16:49.001064 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958a8dc1-0f49-4b2f-afdd-23b4cbab7a30 admin admin] PATH_INFO: `/v3/users/00c145cda7e143299a1c2aff7d408fd7` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:16:49.002233 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958a8dc1-0f49-4b2f-afdd-23b4cbab7a30 admin admin] RBAC: Authorizing `identity:delete_user(user_id=00c145cda7e143299a1c2aff7d408fd7)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:16:49.004161 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958a8dc1-0f49-4b2f-afdd-23b4cbab7a30 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:16:49.180576 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-958a8dc1-0f49-4b2f-afdd-23b4cbab7a30 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '00c145cda7e143299a1c2aff7d408fd7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:49.252824 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-958a8dc1-0f49-4b2f-afdd-23b4cbab7a30 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '00c145cda7e143299a1c2aff7d408fd7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:49.265163 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-958a8dc1-0f49-4b2f-afdd-23b4cbab7a30 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '00c145cda7e143299a1c2aff7d408fd7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:49.265797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-958a8dc1-0f49-4b2f-afdd-23b4cbab7a30 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '00c145cda7e143299a1c2aff7d408fd7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:49.270316 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:16:49.273180 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-958a8dc1-0f49-4b2f-afdd-23b4cbab7a30 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '00c145cda7e143299a1c2aff7d408fd7'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:49.279000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1462/2923] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:16:48 2022] DELETE /identity/v3/users/00c145cda7e143299a1c2aff7d408fd7 => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:16:49.529239 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:16:49.530664 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:16:49.546701 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c7e5ae2-05e5-4e7e-aaa0-75f10a656237 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:16:49.604494 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:16:49.605157 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:16:49.605440 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:16:49.605716 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a admin admin] PATH_INFO: `/v3/projects/abca446946bf4b32a50b47d29e71cba0` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:16:49.606898 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a admin admin] RBAC: Authorizing `identity:delete_project(project_id=abca446946bf4b32a50b47d29e71cba0)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:16:49.608808 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:16:49.707582 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'abca446946bf4b32a50b47d29e71cba0', 'request_id': 'req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:49.713490 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'abca446946bf4b32a50b47d29e71cba0', 'request_id': 'req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:49.801323 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'abca446946bf4b32a50b47d29e71cba0', 'request_id': 'req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:49.813660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a admin admin] The token cache is being invalidate because project abca446946bf4b32a50b47d29e71cba0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81559) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:16:49.813951 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:49.814610 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:16:49.816102 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1462/2924] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:16:49 2022] DELETE /identity/v3/projects/abca446946bf4b32a50b47d29e71cba0 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:16:59.441422 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=3 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:16:59.811206 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=74,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:17:26.080894 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4053e173-48f5-4c38-b282-ff4f8db432bd None None] REQUEST_METHOD: `POST` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:17:26.081415 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4053e173-48f5-4c38-b282-ff4f8db432bd None None] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:17:26.081530 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4053e173-48f5-4c38-b282-ff4f8db432bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:17:26.100699 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.auth.core [None req-4053e173-48f5-4c38-b282-ff4f8db432bd None None] MFA Rules not processed for user `48c4f0c78a364c00a793666eeb5a520c`. Rule list: `[]` (Enabled: `True`). {{(pid=81558) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:17:26.102496 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4053e173-48f5-4c38-b282-ff4f8db432bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:17:26.157305 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1463/2925] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:17:26 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:17:26.466660 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:17:26.468055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:17:26.525297 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5a8b2c-152e-4633-a3fd-1fd641c4244a admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:17:26.528812 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08d3670-7eac-4914-a109-953856664d73 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:17:26.529440 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08d3670-7eac-4914-a109-953856664d73 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:17:26.529718 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08d3670-7eac-4914-a109-953856664d73 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:17:26.529989 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b08d3670-7eac-4914-a109-953856664d73 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:17:26.532301 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b08d3670-7eac-4914-a109-953856664d73 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:17:26.541712 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08d3670-7eac-4914-a109-953856664d73 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:17:26.543114 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b08d3670-7eac-4914-a109-953856664d73 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:17:26.551859 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1463/2926] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:17:26 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:17:27.297641 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4053e173-48f5-4c38-b282-ff4f8db432bd None None] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:17:27.299118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-4053e173-48f5-4c38-b282-ff4f8db432bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:17:27.369495 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4053e173-48f5-4c38-b282-ff4f8db432bd None None] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:17:27.375110 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63817968-6e97-45e1-8859-447af93fc47c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b29f444d56ec49308f7fef16f4d1eea5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:17:27.375731 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63817968-6e97-45e1-8859-447af93fc47c service placement] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:17:27.376175 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63817968-6e97-45e1-8859-447af93fc47c service placement] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:17:27.376455 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63817968-6e97-45e1-8859-447af93fc47c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:17:27.381911 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-63817968-6e97-45e1-8859-447af93fc47c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:17:27.395461 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63817968-6e97-45e1-8859-447af93fc47c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:17:27.397045 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63817968-6e97-45e1-8859-447af93fc47c service placement] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:17:27.407720 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1464/2927] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:17:27 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:17:36.548108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=26 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:17:37.402606 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=57 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:17:41.195592 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08d3670-7eac-4914-a109-953856664d73 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:17:41.197028 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-b08d3670-7eac-4914-a109-953856664d73 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:17:41.276190 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b08d3670-7eac-4914-a109-953856664d73 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:17:41.284194 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4304fa2d-ec59-4ac0-a1c1-a986450bbf96 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:17:41.284811 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4304fa2d-ec59-4ac0-a1c1-a986450bbf96 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:17:41.285108 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4304fa2d-ec59-4ac0-a1c1-a986450bbf96 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:17:41.285463 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4304fa2d-ec59-4ac0-a1c1-a986450bbf96 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:17:41.287180 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-4304fa2d-ec59-4ac0-a1c1-a986450bbf96 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:17:41.301290 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4304fa2d-ec59-4ac0-a1c1-a986450bbf96 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:17:41.303711 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4304fa2d-ec59-4ac0-a1c1-a986450bbf96 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:17:41.315901 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1464/2928] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:17:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:17:51.310370 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=34 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:19:17.833188 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63817968-6e97-45e1-8859-447af93fc47c service placement] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:19:17.834981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-63817968-6e97-45e1-8859-447af93fc47c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:19:17.849350 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63817968-6e97-45e1-8859-447af93fc47c service placement] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:19:17.854849 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a96c87-d50f-4b5f-9974-701052974226 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b29f444d56ec49308f7fef16f4d1eea5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:19:17.855446 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a96c87-d50f-4b5f-9974-701052974226 service placement] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:19:17.855734 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a96c87-d50f-4b5f-9974-701052974226 service placement] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:19:17.856000 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a96c87-d50f-4b5f-9974-701052974226 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:19:17.858082 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-73a96c87-d50f-4b5f-9974-701052974226 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:19:17.870531 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a96c87-d50f-4b5f-9974-701052974226 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:19:17.872118 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a96c87-d50f-4b5f-9974-701052974226 service placement] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:19:17.883625 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1465/2929] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:19:17 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:19:21.698064 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcf5240-5413-42be-b450-ab9f5b368fc6 None None] REQUEST_METHOD: `POST` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:19:21.698648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcf5240-5413-42be-b450-ab9f5b368fc6 None None] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:19:21.698648 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcf5240-5413-42be-b450-ab9f5b368fc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:19:21.720215 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.auth.core [None req-5fcf5240-5413-42be-b450-ab9f5b368fc6 None None] MFA Rules not processed for user `48c4f0c78a364c00a793666eeb5a520c`. Rule list: `[]` (Enabled: `True`). {{(pid=81559) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 04 01:19:21.721365 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-5fcf5240-5413-42be-b450-ab9f5b368fc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:19:21.729121 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1465/2930] 10.222.0.42 () {58 vars in 958 bytes} [Sun Sep 4 01:19:21 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 04 01:19:27.879194 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=6 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:19:31.713786 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=3 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:19:41.207608 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a96c87-d50f-4b5f-9974-701052974226 service placement] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:19:41.219819 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a96c87-d50f-4b5f-9974-701052974226 service placement] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:19:41.225635 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0aa431-13ff-4904-885b-11370839012c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:19:41.226223 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0aa431-13ff-4904-885b-11370839012c service nova] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:19:41.226508 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0aa431-13ff-4904-885b-11370839012c service nova] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:19:41.226786 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0aa431-13ff-4904-885b-11370839012c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:19:41.228479 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-6a0aa431-13ff-4904-885b-11370839012c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:19:41.239515 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0aa431-13ff-4904-885b-11370839012c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:19:41.241104 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0aa431-13ff-4904-885b-11370839012c service nova] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:19:41.250287 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1466/2931] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:19:41 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:19:51.245981 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=6 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:21:14.326385 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcf5240-5413-42be-b450-ab9f5b368fc6 None None] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:21:14.328544 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-5fcf5240-5413-42be-b450-ab9f5b368fc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:21:14.345600 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcf5240-5413-42be-b450-ab9f5b368fc6 None None] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:21:14.351699 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35529779-df37-4e50-9753-72a76aa99820 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '57c88f16c7df40adb917f49804fda21f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'service', 'member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:21:14.357196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35529779-df37-4e50-9753-72a76aa99820 service nova] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:21:14.357196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35529779-df37-4e50-9753-72a76aa99820 service nova] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:21:14.357196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35529779-df37-4e50-9753-72a76aa99820 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:21:14.357196 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-35529779-df37-4e50-9753-72a76aa99820 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:21:14.369344 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35529779-df37-4e50-9753-72a76aa99820 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:21:14.371024 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35529779-df37-4e50-9753-72a76aa99820 service nova] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:21:14.383896 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1466/2932] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:21:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:21:14.479263 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0aa431-13ff-4904-885b-11370839012c service nova] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:21:14.490844 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0aa431-13ff-4904-885b-11370839012c service nova] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:21:14.494617 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7215710a-b1c2-4026-8bbd-ea1719d59231 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:21:14.502997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7215710a-b1c2-4026-8bbd-ea1719d59231 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:21:14.502997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7215710a-b1c2-4026-8bbd-ea1719d59231 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:21:14.502997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7215710a-b1c2-4026-8bbd-ea1719d59231 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:21:14.502997 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-7215710a-b1c2-4026-8bbd-ea1719d59231 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:21:14.509213 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7215710a-b1c2-4026-8bbd-ea1719d59231 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:21:14.510661 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7215710a-b1c2-4026-8bbd-ea1719d59231 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:21:14.519880 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1467/2933] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:21:14 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:21:14.700089 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35529779-df37-4e50-9753-72a76aa99820 service nova] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:21:14.701640 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-35529779-df37-4e50-9753-72a76aa99820 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:21:14.713043 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35529779-df37-4e50-9753-72a76aa99820 service nova] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:21:14.716983 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bf56b4-faa0-45df-8a1d-a06d2120f14a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:21:14.717647 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bf56b4-faa0-45df-8a1d-a06d2120f14a service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:21:14.717930 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bf56b4-faa0-45df-8a1d-a06d2120f14a service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:21:14.718204 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bf56b4-faa0-45df-8a1d-a06d2120f14a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:21:14.720055 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-37bf56b4-faa0-45df-8a1d-a06d2120f14a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:21:14.735022 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bf56b4-faa0-45df-8a1d-a06d2120f14a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:21:14.736444 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bf56b4-faa0-45df-8a1d-a06d2120f14a service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:21:14.749849 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1467/2934] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:21:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:21:15.946216 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7215710a-b1c2-4026-8bbd-ea1719d59231 service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:21:15.955155 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7215710a-b1c2-4026-8bbd-ea1719d59231 service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:21:15.958794 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0eead0-12ab-4f90-b24e-65707bd8db31 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:21:15.959382 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0eead0-12ab-4f90-b24e-65707bd8db31 service neutron] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:21:15.959667 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0eead0-12ab-4f90-b24e-65707bd8db31 service neutron] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:21:15.959949 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0eead0-12ab-4f90-b24e-65707bd8db31 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:21:15.961673 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-ec0eead0-12ab-4f90-b24e-65707bd8db31 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:21:15.975967 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0eead0-12ab-4f90-b24e-65707bd8db31 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:21:15.977497 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0eead0-12ab-4f90-b24e-65707bd8db31 service neutron] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:21:15.989863 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1468/2935] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:21:15 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:21:24.743911 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=12 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:21:25.984742 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=12 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:21:30.004069 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bf56b4-faa0-45df-8a1d-a06d2120f14a service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:21:30.013051 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bf56b4-faa0-45df-8a1d-a06d2120f14a service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:21:30.016702 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f657eb42-f273-4521-a011-6d4905da5f46 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da4f44b86ca94adf959d9a41d915c902', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:21:30.017321 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f657eb42-f273-4521-a011-6d4905da5f46 service neutron] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:21:30.017605 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f657eb42-f273-4521-a011-6d4905da5f46 service neutron] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:21:30.017880 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f657eb42-f273-4521-a011-6d4905da5f46 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:21:30.019615 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f657eb42-f273-4521-a011-6d4905da5f46 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:21:30.082840 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f657eb42-f273-4521-a011-6d4905da5f46 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:21:30.084268 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f657eb42-f273-4521-a011-6d4905da5f46 service neutron] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:21:30.093724 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1468/2936] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:21:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 04 01:21:33.637052 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0eead0-12ab-4f90-b24e-65707bd8db31 service neutron] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:21:33.653508 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0eead0-12ab-4f90-b24e-65707bd8db31 service neutron] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:21:33.658869 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c636bd9c-e81c-4726-81b2-44937ed7697b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b29f444d56ec49308f7fef16f4d1eea5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:21:33.659552 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c636bd9c-e81c-4726-81b2-44937ed7697b service placement] REQUEST_METHOD: `GET` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:21:33.659895 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c636bd9c-e81c-4726-81b2-44937ed7697b service placement] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:21:33.660240 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c636bd9c-e81c-4726-81b2-44937ed7697b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:21:33.662077 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-c636bd9c-e81c-4726-81b2-44937ed7697b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:21:33.675820 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c636bd9c-e81c-4726-81b2-44937ed7697b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:21:33.677450 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c636bd9c-e81c-4726-81b2-44937ed7697b service placement] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:21:33.687895 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1469/2937] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:21:33 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:21:37.357800 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f657eb42-f273-4521-a011-6d4905da5f46 service neutron] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:21:37.359264 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-f657eb42-f273-4521-a011-6d4905da5f46 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:21:37.372039 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f657eb42-f273-4521-a011-6d4905da5f46 service neutron] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:21:37.376894 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222b5a20-ca09-49ee-bf90-4e8d0b7990d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:21:37.377545 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222b5a20-ca09-49ee-bf90-4e8d0b7990d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:21:37.377820 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222b5a20-ca09-49ee-bf90-4e8d0b7990d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:21:37.378089 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222b5a20-ca09-49ee-bf90-4e8d0b7990d6 admin admin] PATH_INFO: `/v3/users/48c4f0c78a364c00a793666eeb5a520c` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:21:37.379197 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222b5a20-ca09-49ee-bf90-4e8d0b7990d6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=48c4f0c78a364c00a793666eeb5a520c)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:21:37.381104 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222b5a20-ca09-49ee-bf90-4e8d0b7990d6 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:21:37.553843 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-222b5a20-ca09-49ee-bf90-4e8d0b7990d6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '48c4f0c78a364c00a793666eeb5a520c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:37.554357 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-222b5a20-ca09-49ee-bf90-4e8d0b7990d6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '48c4f0c78a364c00a793666eeb5a520c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:37.634316 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-222b5a20-ca09-49ee-bf90-4e8d0b7990d6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '48c4f0c78a364c00a793666eeb5a520c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:37.646625 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-222b5a20-ca09-49ee-bf90-4e8d0b7990d6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '48c4f0c78a364c00a793666eeb5a520c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:37.651127 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-222b5a20-ca09-49ee-bf90-4e8d0b7990d6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '48c4f0c78a364c00a793666eeb5a520c'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:37.659549 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1469/2938] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:21:37 2022] DELETE /identity/v3/users/48c4f0c78a364c00a793666eeb5a520c => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:21:37.901876 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c636bd9c-e81c-4726-81b2-44937ed7697b service placement] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:21:37.912844 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c636bd9c-e81c-4726-81b2-44937ed7697b service placement] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:21:37.971607 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c5115c-d94e-4658-8749-e940a9cefe6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:21:37.972238 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c5115c-d94e-4658-8749-e940a9cefe6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:21:37.972521 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c5115c-d94e-4658-8749-e940a9cefe6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:21:37.972797 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c5115c-d94e-4658-8749-e940a9cefe6e admin admin] PATH_INFO: `/v3/projects/2506b96ac13541e6a7a47c5aedca67e5` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:21:37.973947 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c5115c-d94e-4658-8749-e940a9cefe6e admin admin] RBAC: Authorizing `identity:delete_project(project_id=2506b96ac13541e6a7a47c5aedca67e5)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:21:37.976422 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c5115c-d94e-4658-8749-e940a9cefe6e admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:21:38.145952 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-47c5115c-d94e-4658-8749-e940a9cefe6e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2506b96ac13541e6a7a47c5aedca67e5', 'request_id': 'req-47c5115c-d94e-4658-8749-e940a9cefe6e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:38.151429 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-47c5115c-d94e-4658-8749-e940a9cefe6e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2506b96ac13541e6a7a47c5aedca67e5', 'request_id': 'req-47c5115c-d94e-4658-8749-e940a9cefe6e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:38.157633 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-47c5115c-d94e-4658-8749-e940a9cefe6e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2506b96ac13541e6a7a47c5aedca67e5', 'request_id': 'req-47c5115c-d94e-4658-8749-e940a9cefe6e'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:38.233652 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-47c5115c-d94e-4658-8749-e940a9cefe6e admin admin] The token cache is being invalidate because project 2506b96ac13541e6a7a47c5aedca67e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:21:38.233940 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-47c5115c-d94e-4658-8749-e940a9cefe6e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:38.234563 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-47c5115c-d94e-4658-8749-e940a9cefe6e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:38.236063 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1470/2939] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:21:37 2022] DELETE /identity/v3/projects/2506b96ac13541e6a7a47c5aedca67e5 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:21:38.240279 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222b5a20-ca09-49ee-bf90-4e8d0b7990d6 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:21:38.242598 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-222b5a20-ca09-49ee-bf90-4e8d0b7990d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:21:38.255769 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222b5a20-ca09-49ee-bf90-4e8d0b7990d6 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:21:38.313182 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37044197-3b31-4e5a-856e-82a744f10a39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:21:38.313817 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37044197-3b31-4e5a-856e-82a744f10a39 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:21:38.314095 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37044197-3b31-4e5a-856e-82a744f10a39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:21:38.314362 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37044197-3b31-4e5a-856e-82a744f10a39 admin admin] PATH_INFO: `/v3/users/eaaaa268a2664111b8a25f368bb354fc` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:21:38.315518 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37044197-3b31-4e5a-856e-82a744f10a39 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eaaaa268a2664111b8a25f368bb354fc)` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:21:38.317525 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37044197-3b31-4e5a-856e-82a744f10a39 admin admin] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:21:38.322822 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:21:38.533541 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-37044197-3b31-4e5a-856e-82a744f10a39 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eaaaa268a2664111b8a25f368bb354fc'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:38.534217 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-37044197-3b31-4e5a-856e-82a744f10a39 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eaaaa268a2664111b8a25f368bb354fc'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:38.599634 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-37044197-3b31-4e5a-856e-82a744f10a39 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eaaaa268a2664111b8a25f368bb354fc'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:38.612688 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-37044197-3b31-4e5a-856e-82a744f10a39 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eaaaa268a2664111b8a25f368bb354fc'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:38.613391 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.notifications [None req-37044197-3b31-4e5a-856e-82a744f10a39 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eaaaa268a2664111b8a25f368bb354fc'} {{(pid=81559) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:38.625848 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1470/2940] 10.222.0.42 () {58 vars in 1224 bytes} [Sun Sep 4 01:21:38 2022] DELETE /identity/v3/users/eaaaa268a2664111b8a25f368bb354fc => generated 0 bytes in 386 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:21:38.893340 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c5115c-d94e-4658-8749-e940a9cefe6e admin admin] Authenticating user token {{(pid=81558) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:21:38.895546 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.fernet_utils [None req-47c5115c-d94e-4658-8749-e940a9cefe6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81558) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:21:38.908732 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c5115c-d94e-4658-8749-e940a9cefe6e admin admin] Validating token access rules against request {{(pid=81558) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:21:38.968149 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41dd9bc4-a3fb-4b8c-88cd-8e62c0a57066 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '01782590dd294bd488a6d5c08a043ce1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'f25649add4b1426590abafbe586fc1fa', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81558) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:21:38.968780 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dd9bc4-a3fb-4b8c-88cd-8e62c0a57066 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:21:38.969056 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dd9bc4-a3fb-4b8c-88cd-8e62c0a57066 admin admin] SCRIPT_NAME: `/identity` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:21:38.969372 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dd9bc4-a3fb-4b8c-88cd-8e62c0a57066 admin admin] PATH_INFO: `/v3/projects/fa1ba3942811492fb9ef388575baa4e2` {{(pid=81558) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:21:38.970462 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41dd9bc4-a3fb-4b8c-88cd-8e62c0a57066 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa1ba3942811492fb9ef388575baa4e2)` {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:21:38.972342 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41dd9bc4-a3fb-4b8c-88cd-8e62c0a57066 admin admin] RBAC: Authorization granted {{(pid=81558) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:21:38.994156 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:21:39.104429 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-41dd9bc4-a3fb-4b8c-88cd-8e62c0a57066 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa1ba3942811492fb9ef388575baa4e2', 'request_id': 'req-41dd9bc4-a3fb-4b8c-88cd-8e62c0a57066'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:39.110030 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-41dd9bc4-a3fb-4b8c-88cd-8e62c0a57066 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa1ba3942811492fb9ef388575baa4e2', 'request_id': 'req-41dd9bc4-a3fb-4b8c-88cd-8e62c0a57066'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:39.118745 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-41dd9bc4-a3fb-4b8c-88cd-8e62c0a57066 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa1ba3942811492fb9ef388575baa4e2', 'request_id': 'req-41dd9bc4-a3fb-4b8c-88cd-8e62c0a57066'} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:39.193477 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-41dd9bc4-a3fb-4b8c-88cd-8e62c0a57066 admin admin] The token cache is being invalidate because project fa1ba3942811492fb9ef388575baa4e2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81558) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 04 01:21:39.193770 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-41dd9bc4-a3fb-4b8c-88cd-8e62c0a57066 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:39.195103 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG keystone.notifications [None req-41dd9bc4-a3fb-4b8c-88cd-8e62c0a57066 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81558) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 04 01:21:39.196665 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: [pid: 81558|app: 0|req: 1471/2941] 10.222.0.42 () {58 vars in 1233 bytes} [Sun Sep 4 01:21:38 2022] DELETE /identity/v3/projects/fa1ba3942811492fb9ef388575baa4e2 => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 04 01:21:48.622120 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=23,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:21:49.251675 dv-cinder-master-3-iscsi devstack@keystone.service[81558]: DEBUG dbcounter [-] [81558] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81558) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 04 01:24:24.872855 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37044197-3b31-4e5a-856e-82a744f10a39 admin admin] Authenticating user token {{(pid=81559) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 04 01:24:24.876235 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-37044197-3b31-4e5a-856e-82a744f10a39 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:24:25.011130 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37044197-3b31-4e5a-856e-82a744f10a39 admin admin] Validating token access rules against request {{(pid=81559) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 04 01:24:25.070569 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc704f1-ad31-4b7a-b9e8-6c612d1e251b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b29f444d56ec49308f7fef16f4d1eea5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b1f56314d44d4faaa88e735ea196783f', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81559) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 04 01:24:25.071188 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc704f1-ad31-4b7a-b9e8-6c612d1e251b service placement] REQUEST_METHOD: `GET` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 04 01:24:25.071471 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc704f1-ad31-4b7a-b9e8-6c612d1e251b service placement] SCRIPT_NAME: `/identity` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 04 01:24:25.071748 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc704f1-ad31-4b7a-b9e8-6c612d1e251b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81559) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 04 01:24:25.074027 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.fernet_utils [None req-8bc704f1-ad31-4b7a-b9e8-6c612d1e251b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81559) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 04 01:24:25.086322 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc704f1-ad31-4b7a-b9e8-6c612d1e251b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 04 01:24:25.087894 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc704f1-ad31-4b7a-b9e8-6c612d1e251b service placement] RBAC: Authorization granted {{(pid=81559) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 04 01:24:25.098513 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: [pid: 81559|app: 0|req: 1471/2942] 10.222.0.42 () {60 vars in 1365 bytes} [Sun Sep 4 01:24:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 04 01:24:35.094213 dv-cinder-master-3-iscsi devstack@keystone.service[81559]: DEBUG dbcounter [-] [81559] Writing DB stats keystone:SELECT=89 {{(pid=81559) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}